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APPENDIX A. IMPLEMENTATION OF THE 2011 PELL GRANT EXPERIMENTS 


This appendix provides additional details about the 2011 Pell Grant experiments described on p. 2 of 
the report. In particular, the appendix describes the process for approving schools and programs to 
be included in the experiments, how schools implemented the experiments, the characteristics of 
study schools and eligible programs, and the amount of experimental Pell Grant funds disbursed 
over the study period. This information may help interpretation of the evaluation results and identify 
issues policymakers might consider if they make the experimental waivers of Pell Grant eligibility 
rules universally available. 


A.1. Approving schools and programs to be included in the experiments 


The Office of Federal Student Aid (FSA) announced the 2011 Pell Grant experiments to the public, 
postsecondary institutions potentially eligible for the experiments in October 27, 2011.' Schools had 
to complete a series of steps to be approved to participate in any experiment under the Experimental 
Sites Initiative (Exhibit A.1). These steps are as follows: 


e Submit a letter of application to FSA indicating an intent to participate and the ability to report 
required data (Step 1). 


e Undergo screening for past compliance, to ensure the schools are in good standing in 
administering federal aid under Title IV Student Assistance programs (Step 2). 


e If invited to participate (Step 3), send a signed Program Participation Agreement (PPA) 
Amendment back to FSA (Step 4). The PPA identifies the specific waivers being granted under the 
experiment and the reporting requirements for participation. Only after FSA approves and 
countersigns the form are schools officially approved to participate in an experiment. 


e Schools interested in participating in Experiment 2 had some additional steps. Each of the very 
short-term programs they wanted to make eligible for a Pell Grant under the experiment had to 
be approved by FSA. Schools had to submit a Program Approval Form along with documentation 
that the program itself had been approved by the schools’ accrediting body or its state agency, or 
both (Step 5). 


Exhibit A.1. School and program approval process 


School approval Program approval 
Experiments 1 and 2 Experiment 2 only 
1 2 3 4 5 


School submits School screened for School sent School submits School submits the 
letter of interest/ past compliance invitation to a signed PPA to Approval of Short 


application to FSA participate from the FSA’s School Term Programs Form 
FSA Participation with supporting 
Division documentation 


FSA = Federal Student Aid, PPA = Program Participation Agreement 


' https://ifap.ed.gov/sites/default/files/attachments/fregisters/FR102711ExperimentalSitesInitiative.pdf. 


After the initial call for applications, FSA conducted additional outreach three times (in January 2013, 
February 2013, and August 2013) via electronic communications to financial aid professionals 
because schools were initially slow to apply.” Despite the multistep process, 84 schools sought and 
obtained approval to participate in one or both experiments between the initial announcement and 
March 31, 2017, when the experiments were closed to new schools. Most were approved in 2013 
(Exhibit A.2). However, among the schools that were approved for Experiment 2, only 35 schools 
went on to have a very short-term program approved by FSA, suggesting possible challenges related 
to identifying programs or obtaining additional approvals or accreditation required for these 
programs to be part of the experiment. 


Exhibit A.2. Number of schools approved for Experiments 1 and 2, by year 
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SOURCE: Office of Federal Student Aid, Random Assignment System. 


NOTE: Sixty-eight schools were approved (submitted a signed Program Participation Agreement Amendment) for each experiment by March 
31, 2017; four schools were omitted because of missing data on the year in which they were approved. Due to the small number of schools 
approved in 2014-2016, the exhibit combines these years to protect respondent confidentiality, in accordance with National Center for 
Education Statistics statistical standards. 


A.2. School implementation of the 2011 Pell Grant experiments 


After obtaining approval from FSA, school financial aid offices were responsible for recruiting 
students and determining their eligibility for the experiments. Because the experiments targeted 
students not ordinarily eligible for Pell Grants, schools had to develop their own procedures to find 
and flag them for the experiments, rather than process their aid packages in the usual manner. 
Students eligible for the experiments were those who: 


? The initial announcement (https://www.federalregister.gov/documents/2011/10/27/2011-27880/postsecondary-educational-institutions-invited- 
to-participate-in-experiments-under-the-experimental) specified that schools that applied by December 12, 2011, would receive priority to be 
considered for participation. An electronic communication to financial aid professionals in January 2013 extended the deadline to February 28, 
2013. A February 2013 electronic communication further extended the deadline to April 30, 2013. Finally, an August 2013 electronic 
communication reopened the application process without a closing date. 


e Met all other Pell Grant eligibility criteria (for example, demonstrating financial need, being a 
U.S. citizen or eligible noncitizen)°, 


e Were un- or underemployed’, 

e Expressed interest in an eligible occupational training program, 
e Were entering the program for the first time, and 

e For Experiment 1, had already obtained a bachelor’s degree. 


In addition, schools were encouraged to engage local workforce partners to advertise the availability 
of the experimental Pell Grants for short occupational training and to recruit new students, but were 
not given supplemental funds to cover the additional costs associated with carrying out this or any 
other aspect of the experiments. 


Exhibit A.3. Number of approved schools and study schools 


Number of approved schools* Number of study schools? 


Experiment 1 68 35 
Experiment 2 68 28 
Both experiments 52 17 
Total 84 46 


Source: Office of Federal Student Aid. 


Note: The sum of Experiment 1 and Experiment 2 schools does not equal the total number of schools, because some schools were in both 
experiments and included in the counts for each experiment separately. 


* Approved schools are those approved by FSA for participation in Experiment 1 and/or 2 that submitted a signed Program Participation 
Agreement amendment. 


» Study schools are those approved to participate that identified at least one eligible participant. 


Between November 2012 and March 2017, only about half of the schools approved by FSA went on to 
identify eligible students for the study (Exhibit A.3), taking, on average, nearly nine months from 
approval to identify their first eligible student and begin participating in either experiment (Exhibit 
A.4). Challenges with recruiting new students, with which school financial aid offices had little to no 
experience, and difficulties with identifying students not traditionally eligible for Title IV aid may 
have contributed to delays in schools’ participating or in their decision to ultimately not participate 
in the experiments. 


3 https://studentaid.gov/understand-aid/eligibility/requirements 


4 The Office of Federal Student Aid did not offer additional guidance to schools on how to determine whether students were un- 
or underemployed. 


Exhibit A.4. Time between school approval by FSA and beginning of schools’ participation in 
the experiments 
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SOURCE: Office of Federal Student Aid, Random Assignment System. 

NOTE: The time elapsed between school approval and a school’s participation is calculated as the number of months between when the 
school’s Program Participation Agreement amendment was entered into FSA’s Postsecondary Education Participants System and when they 
identified (and randomly assigned) their first eligible student for Experiment 1 or 2. Of the 68 schools approved for Experiment 1, 32 schools 
had begun participating by March 31, 2017; three schools are omitted because of missing data on the year in which they were approved. Of the 
68 schools approved for Experiment 2, 27 schools had begun participating by March 31, 2017; one school is omitted because of missing data on 
the year in which it was approved. On average, 8.75 months elapsed between approval and when schools began participating in Experiment 1, 
and 8.85 months elapsed between approval and when schools began participating in Experiment 2. For Experiment 2, 3 of the 40 schools that 
did not identify an eligible student and begin participating in the experiment had a very short-term program approved by the Office of Federal 
Student Aid. 


Schools that ended up participating in either experiment, which this report refers to as study 
schools, tended to identify relatively low numbers of students. Study schools identified 13 students 
for Experiment 1 and 83 students for Experiment 2, on average, but most schools identified fewer 
than 10 students for Experiment 1 and fewer than 50 students for Experiment 2 (Exhibits A.5 and 
A.6). Overall, study schools identified 2,914 eligible students (not shown). 


Exhibit A.5. Number of students identified for Experiment 1, per study school 
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SOURCE: Random Assignment System. 
NOTE: This exhibit contains information from the 35 schools that identified eligible students for Experiment 1 only between the time each 
school began participating in the experiment by March 31, 2017. 


Exhibit A.6. Number of students identified for Experiment 2, per study school 
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SOURCE: Random Assignment System. 
NOTE: This exhibit contains information from the 28 schools that identified eligible students for Experiment 2 only between the time each 
school began participating in the experiment by March 31, 2017. 


Because FSA had a limited amount of experimental Pell Grant funds to make available, schools used 
a lottery-like process to allocate the funds. Eligible students were randomly assigned to be offered 
experimental Pell Grants (treatment group) or to not be offered experimental Pell Grants (control 
group). Schools wanted to be able to determine in real time (with little delay) and on a rolling basis 


whether students would be offered experimental Pell funds or not, consistent with how they 
otherwise prepare students’ financial aid awards. In addition, they wanted to award experimental 
Pell Grants to as many students as possible. To accommodate those school preferences, the study 
had financial aid staff enter minimal information about each eligible student into a secure web-based 
Random Assignment System that selected a student for the offer of experimental Pell Grants 60 
percent of the time and immediately responded with information about the group to which the 
student was assigned. See Appendix B for more details about the random assignment process. 


A.3. Characteristics of study schools and eligible programs 


Study schools 


The more the study schools look like schools nationally, the more relevant the study’s findings are to 
what could happen if the experiments become permanent federal aid policy. The study’s 46 schools 
were not representative of all postsecondary schools in the United States (Exhibit A.7), suggesting 
that the study’s findings may not hold if the experiments become permanent policy. Whereas the 
majority of participating schools (72 percent) were public, two-year institutions, only 16 percent of 
postsecondary schools in the U.S. fall under this category. This difference is consistent with the 
experiments’ focus on short-term and very short-term occupational training programs. However, 
participating schools were also more concentrated in the southeast region of the U.S. and more likely 
to be located in urban centers. 


Exhibit A.7. Characteristics of study schools compared with all postsecondary schools in the United 
States 


All U.S. postsecondary 


Characteristic Study schools? rye sore) kg 
Type of school 

Public, two-year 72% 16% 
Public, less than two-year 13% 4% 
Other 15% 80% 
Regional location 

Southeast (AL, AR, FL, GA, KY, LA, MS, NC, SC, TN, VA, WV) 46% 25% 
Great Lakes (IL, IN, MI, OH, WI) 17% 15% 
Far West (AK, CA, HI, NV, OR, WA) 15% 14% 
Mideast (DE, DC, MD, NJ, NY, PA) 13% 17% 
Southwest (AZ, NM, OK, TX) 9% 11% 
Plains (IA, KS, MN, MO, NE, ND, SD) 0% 9% 
New England (CT, ME, MA, NH, RI, VT) 0% 6% 
Rocky Mountains (CO, ID, MT, UT, WY) 0% 3% 
Urbanicity 

Urban 57% 48% 
Suburb 22% 30% 
Rural 13% 9% 
Town 9% 14% 
Total 46 6,319 


SOURCE: Integrated Postsecondary Education Data System 2018. 

NOTE: “Other” includes public four-year schools and private for-profit and not-for-profit schools. 

@ Study schools are those approved by FSA to participate in Experiment 1 and/or Experiment 2 that identified at least one eligible participant by 
March 31, 2017. 


» All postsecondary schools in the U.S. refers to Title IV institutions, which are those with a written agreement with ED that allows the 
institution to participate in any of the Title IV federal student financial assistance programs. 


Beyond location, the participating schools stand out in terms of size, outcomes, and who they serve. 
Even relative to similar schools in their states, study schools tended to be either smaller or larger 
(Exhibit A.8), have similar graduation rates (Exhibit A.9), and have a higher percentage of Pell Grant 
recipients (Exhibit A.10). 


Exhibit A.8. Study schools’ total enrollment as a share of the average total enrollment for 
similar schools in the state 
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SOURCE: Integrated Postsecondary Education Data System 2014. 

NOTE: This exhibit includes all 46 study schools (those approved to participate in Experiment 1 and/or Experiment 2 by FSA that identified at 
least one eligible participant by March 31, 2017). Each study school is compared to the state average for similar schools (that is, public two-year 
schools are compared to the statewide average for public two-year schools, and so on). The average total enrollment for schools in the 20 
states in which these 46 schools operated was 7,635 students. The average total enrollment of the 46 study schools is 9,940. These data reflect 
enrollment rates for the 2014-2015 academic year. 


Exhibit A.9. Study schools’ graduation rate as a share of the average graduation rate for similar 
schools in the state 
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Graduation rate as a share of the state average 


SOURCE: Integrated Postsecondary Education Data System 2014. 

NOTES: This exhibit includes 45 study schools (those approved to participate by FSA that identified at least one eligible participant by March 
31, 2017); one school is omitted because of missing graduation rate data. Each participating school is compared to its state average for similar 
schools (that is, public two-year schools are compared to the statewide average for public two-year schools, and so on). The average graduation 
rate for schools in the 20 states in which the participating schools operated was 36 percent. The average graduation rate for the 45 
participating schools is 37 percent. These data reflect graduation rates for the 2014-2015 academic year. 


Exhibit A.10. Study schools’ regular Pell Grant recipient rate as a share of the average Pell Grant 
recipient rate for similar schools in the state 
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SOURCE: Integrated Postsecondary Education Data System 2014. 

NOTE: This exhibit includes 45 study schools (those approved by FSA to participate that identified at least one eligible participant by March 31, 
2017); four schools are omitted because of missing Pell Grant recipient rate data. Each participating school is compared to its state average for 
similar schools (that is, public two-year schools are compared to the statewide average for public two-year schools, etc.). The average Pell 
Grant recipient rate (percentage of students receiving Pell funds) for the 20 states in which the participating schools operated is 44 percent. 
The average Pell Grant recipient for the 45 participating schools is 48 percent. These data reflect regular Pell Grant recipient rates for the 2014- 
2015 academic year. 


Eligible programs 


The key goal for the experiments was to train more un- and underemployed individuals for careers in 
high-demand occupations, as the stepping-stone to higher employment and earnings. Programs 
eligible under the experiments were expected to meet a local or regional workforce need, but 
schools had complete discretion over how to make that determination. For Experiment 1, eligible 
students could enroll in a wide range of programs lasting up to one year that were already eligible for 
Pell Grants at each approved school. However, for Experiment 2, only specific programs were 
approved by FSA to provide very short-term training options. 


As part of the process to determine student eligibility, students indicated the program of study that they 
wanted to enter. Almost 50 percent of Experiment 2 students indicated an intention to enroll in 
transportation and materials moving programs; 35 percent intended to enroll in programs to prepare 
them for health professions and careers in related clinical sciences (Exhibit A.11). On average, the 
duration of programs in which Experiment 2 students intended to enroll was 11.4 weeks (not shown).° 


Because schools did not report how they determined whether programs met a local or regional 
workforce need, the study examined the proportion of programs associated with high-demand 
occupations, as opposed to occupations with few openings and little expected growth. In both 
experiments, over half of study participants intended to enroll in a program in a high-demand 
occupation in their state: 53 percent of Experiment 1 students and 58 percent of Experiment 2 
students (not shown). Appendix B describes the methods the study used to determine whether a 
program was associated with an occupation in high demand. 


Exhibit A.11. Percentage of students in Experiment 2 who expressed interest in FSA-approved 
very short-term occupational training programs, by field 


Transportation and materials moving 47% 
Health professions and related clinical sciences 
Security and protective services 


Mechanic and repair technologies/technicians 


Other 


[ T T T T 1 
0% 20% 40% 60% 80% 100% 
Percentage of Experiment 2 students 


SOURCE: Random Assignment System. 

NOTE: This exhibit contains information for the 2,270 students in the Experiment 2 analysis sample (see Appendix B for information on the 
formation of the analysis sample). Information on the intended program is missing for 14 students. The exhibit reports two-digit Classification 
of Instructional Program codes associated with the FSA-approved programs in which Experiment 2 students expressed interest at the time of 
random assignment. These programs were two to four months (8-15 weeks) in duration. 


5 Because the programs in which Experiment 1 students intended to enroll were not subject to FSA approval, schools did not 
report data on these programs consistently. Because of the large amount of missing data for Experiment 1 programs, the study 
does not report on the type or duration of these programs. 


A.4, Amount of Pell Grants disbursed under the 2011 Pell Grant experiments 


The amount of experimental Pell Grant funds a student could receive, which was determined by the 
study schools, was based on the schools’ cost of attendance, the student’s expected family financial 
contribution, and enrollment status (the number of credit or clock hours for which the student 
enrolled)—a process similar to determining the amounts awarded for regular Pell Grants. For 
example, for the 2014-2015 award year, the maximum Pell Grant available was $5,730.° This amount 
could be lower for students with a lower cost of attendance, higher expected family financial 
contribution, or who were enrolled less than full time.’ Schools participating in Experiment 2 
prorated the amount of experimental Pell Grants awarded based on the duration of very short 
programs. 


The 2011 Pell Grant experiments cost approximately $1.5 million ($1,752 per student), on average, 
over six years (2012-2018; Exhibit A.12).8 The share of students who were offered experimental Pell 
Grants and went on to use (“take up”) the grants offered was 55 percent overall (68 percent among 
Experiment 1 students and 53 percent among Experiment 2 students; Exhibit A.12).° On average, 
Experiment 1 students who did use the grants received $3,556, and Experiment 2 students who used 
the grants received $1,315 (Exhibit A.12). These amounts were reported by study schools and could 
cover more than one program and award year. 


Exhibit A.12. Experimental Pell Grant take-up rates and disbursement amounts 


Experiment 1 Experiment 2 Overall 
Number of students offered experimental Pell Grant 254 1,363 1,617 
Number of students offered experimental Pell Grant who 170 705 875 
used it 
Total amount of experimental Pell Grants disbursed $608,009 $924,648 $1,532,657 
Take-up rate (number of students who used the grant 67% 52% 54% 
divided by the number of students offered experimental 
Pell Grants) 
Average experimental Pell Grant amount disbursed (total $3,577 $1,312 $1,752 
amount disbursed divided by the number of students 
who used the experimental Pell Grant) 


Source: School records. 


6 Source: Information for Financial Aid Professionals Dear Colleague Letter GEN-14-01. Available at https://ifap.ed.gov/dear- 
colleague-letters/01-31-2014-gen-14-01-subject-2014-2015-federal-pell-grant-payment-and. 


7 Source: https://ifap.ed.gov/federal-student-aid-handbook/07-11-2014-volume-3-calculating-awards-packaging 


8 Students were identified on a rolling basis between November 2012 and March 2017. The last program completion date 
observed in school records was August 2018. 


° Analyses exploring which types of students were likely to take up the offer of an experimental Pell Grant are described in 
Appendix B and findings are reported in Appendix C. 
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APPENDIX B. STUDY DESIGN AND ANALYTIC APPROACH 


This appendix provides additional details on the study design for those interested in the research 
methods used to assess the experiments’ implementation and effects. The appendix describes the 
research questions and logic model underlying the study, the random assignment process that 
determined who was offered an experimental Pell Grant, how the analytic sample was identified, and 
the outcome measures and data sources used. In addition, it provides technical details on the 
analytic approaches used to estimate the effects of the experiments. 


B.1. Research questions 


This study was designed to provide evidence on the effects of making Pell Grants available to 
individuals who had already earned a bachelor’s degree (Experiment 1) and individuals interested in 
very short-term occupational training programs (Experiment 2). Given the time frame for the study, 
the research questions focused on assessing the experiments’ effects on short-term outcomes and 
describing the use of experimental Pell funds. Specifically, the study asked the following questions: 


1. Did offering experimental Pell Grants increase enrollment in and completion of postsecondary 
programs in study schools? Because experimental Pell Grants could only be used in schools 
approved to participate in the experiments, the study focused on enrollment and completion in 
these study schools only. The study’s focus was enrollment in any program because students 
could have entered other programs in these schools that might also be beneficial to them. 


2. What percentage of students who were offered experimental Pell Grants used them? 
Understanding the share of students who actually used the experimental Pell Grants offered to 
them is important for policymakers considering permanently broadening access to Pell Grants. 


The study also analyzed additional exploratory questions to better understand the effects of the 
experiments. These exploratory analyses aimed to answer the following questions: 


1. Would the effects on enrollment and completion be different if the analysis were extended 
beyond the study schools, or beyond the study’s completion window? Enrollment and completion 
in a broader set of schools may better capture the behavior of students who were not offered Pell 
Grants. For example, the offer of experimental Pell Grants could lead students to enroll in and 
complete programs at the study schools; conversely, students who were not offered 
experimental Pell Grants might enroll in and complete programs at other, less expensive 
colleges. The study also explored whether the effects on enrollment and completion could be 
affected by the study’s completion window (for example, students not offered experimental Pell 
Grants could have enrolled in longer programs that would have taken longer to complete). 


2. Did offering experimental Pell Grants affect the types of programs that students enrolled in or 
completed? Exploring whether offering experimental Pell Grants moved students into short 
programs associated with high-demand occupations or away from longer programs, such as an 
associate’s degree, would give insight into the potential employment benefits of extending Pell 
Grant eligibility in this way. 


3. Did offering experimental Pell Grants affect students’ use of federal student loans? 
Understanding how having access to experimental Pell grants affected students’ needs for 
additional financial aid is important for policymakers considering whether to make the 
experimental waivers permanent. 


B.2. Random assignment and identification of the analytic sample 


Random assignment was used to determine which students would be offered experimental Pell 
Grants. Random assignment ensures that students who are offered experimental Pell Grants are 
statistically equivalent to students who did not receive an offer, such that any differences in their 
later outcomes can be interpreted as the result of being offered an experimental Pell Grant. It is the 
most rigorous study design available. Random assignment also provided a fair way to determine who 
was offered the experimental Pell Grants since a limited amount of funds were available to support 
students interested in the pilot. 


The web-based Random Assignment System (RAS) developed for the study assigned students 
randomly to either be offered experimental Pell Grants (treatment group) or not offered the grants 
(control group), with a 60 percent probability that a student would be assigned to receive an offer. 
Students were randomly assigned within each experiment for which they were eligible,'° and within 
their study school and occupational training program of interest. This type of random assignment 
(known as stratified random assignment) ensured that the study schools and programs were 
proportionally represented in each experiment. 


In total, study schools identified and randomly assigned 2,914 eligible students for the experiments 
between November 2012 and March 2017: 437 students for Experiment 1, 2,399 for Experiment 2, and 
78 students who were interested in and eligible for both experiments (Exhibit B.1). After random 
assignment, the RAS sent an automatic email to study participants informing them of their inclusion 
in the study and telling them how to revoke consent if they did not want to be part of the study." 


Some of the students who were randomly assigned had to be excluded from the analytic sample used 
to answer the study’s research questions. Students were excluded from the analytic sample if they (1) 
revoked consent (4 people), (2) were determined by the schools to be ineligible for the study after 

random assignment (4 people),” or (3) were subject to a technical problem in the random assignment 


‘© Students participating in both Experiment 1 and Experiment 2 were treated as if they were part of a third experiment; to 
minimize crossover, they were randomly assigned to either receive experimental Pell funds or not for both experiments. 


"Tf the eligible student was determined to be a minor, the RAS required contact information for a parent or guardian, who was 
then notified about participation in the study and how to opt out. 


® Schools determined these four students to be ineligible because they either already had a bachelor’s degree but were assigned 
to Experiment 2 only or never provided verification for eligibility. 
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process (151 people). One hundred fifty-nine students were randomly assigned but excluded from 
analyses: 23 in Experiment 1, 129 in Experiment 2, and 7 students who participated in both 
experiments (Exhibit B.1). 


Exhibit B.1. Number of students randomly assigned and included in the analytic sample, by 
experiment 


Number of students randomly assigned Number of students in the analytic sample* 
Offered Not offered Offered Not offered 
experimental experimental experimental experimental 

Experiment Pell Grant Pell Grant Total Pell Grant Pell Grant Total 
Experiment 1 277 160 437 254 160 414 
Experiment 2 1,488 911 2,399 1,363 907 2,270 
Both experiments 54 24 78 47 24 71 
Total 1,819 1,095 2,914 1,664 1,091 2,755 


SOURCE: Random Assignment System. 


@The analytic sample includes the 414 students who participated in Experiment 1 only and the 2,270 students who participated in Experiment 2 
only. A sensitivity analysis (described in detail in Appendix D) includes the 71 students who participated in both experiments. 


To estimate the effect of each specific type of Pell Grant eligibility waiver, the analyses for each 
experiment were conducted separately and the main analysis sample was restricted to the 414 
students who participated in Experiment 1 only and the 2,270 students who participated in 
Experiment 2 only (Exhibit B.1). Although the 71 students who participated in both experiments were 
excluded from the main analyses, sensitivity analyses that added them back into each experiment led 
to the same findings described in the report. (See Appendix D for more about the sensitivity 
analyses.) Because the random assignment process grouped students eligible for both experiments 
into a separate experiment (or stratum) before conducting random assignment, excluding them does 
not violate random assignment and does not contribute to sample attrition. 


Attrition 


Sample attrition occurs when an analytic sample excludes study participants that were already 
randomly assignment to an experiment. Attrition can introduce bias in study findings if different 
types of students are more likely than others to be excluded from the analytic sample, therefore 
making the treatment and control groups dissimilar. To determine whether attrition represented a 
threat to the validity of findings, the study assessed the level of attrition overall and the difference 
between the treatment and control groups (known as the differential attrition rate) for each 


3 Two types of errors occurred. First, the RAS program had built-in code so that once the number of participants assigned to the 
control group within a school reached a certain threshold, all subsequent participants at that school would be assigned to the 
treatment group. This rule was created for studies where this option was desired, which was not the case in this study; therefore, 
the threshold was supposed to be set to 1,000 so that in practice it would never be reached and the rule would not be triggered. 
By mistake, this threshold was not set for three participating schools, resulting in all study participants at these schools being 
assigned to the treatment group (N = 144). Second, when the social security number (SSN) or an alternate ID for a potential new 
study participant was entered into the system, the system checked whether that information matched that of an existing study 
participant. If it did, the student was assigned to the experimental group to which the existing study participant was assigned, 
assuming they were the same person. This rule made sense when using SSNs, because SSNs are unique. However, participating 
schools did not always use the alternate as a unique identifier, which meant that multiple individuals could share the same ID. 
Eight participants in seven schools were affected by this error, one of whom was also subject to the first error. Of the 151 
participants with a random assignment error, 7 were in both experiments. 
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experiment (Exhibit B.2). The combination of overall and differential attrition in each experiment 
represents tolerable levels that are not expected to bias the study’s results." 


Exhibit B.2. Overall and differential attrition rates for Experiment 1 and Experiment 2 


Attrition rate in offered 


Attrition rate in not 


experimental Pell Grant offered experimental Whicauaney| Overall attrition 
Experiment Fa weyt yo) Pell Grant group attrition rate rate 
Experiment 1 8.3% 0.0% 8.3% 5.3% 
Experiment 2 8.4% 0.4% 8.0% 5.4% 


SOURCE: Random Assignment System. 


Another way to assess the potential threat of attrition to the study’s validity is to determine whether 
the students excluded from the analysis were different from the rest of the sample. As shown in 
Exhibit B.3, the analytic samples were similar to the full sample for each experiment. 


The study also compared the demographic characteristics of students in the analytic sample to those 
of all postsecondary students in the United States to understand how representative the study’s 
findings might be if experimental Pell Grants were made available nationwide. Students in 
Experiment 1 were similar to all postsecondary students in the United States, on average, in terms of 
gender: the majority were female, and approximately 35 percent were male. Experiment 2 students, 
on the other hand, were more likely to be male than all postsecondary students nationally. In terms 
of age, students in both experiments tended to be older than all postsecondary students nationally 
(Exhibit B.3). 


Exhibit B.3. Characteristics of all postsecondary students in the United States and students in the 
study’s full and analytic sample 


Experiment 1 


Experiment 2 


All postsecondary 

Full sample Analytic sample Full sample Analytic sample students in the 
(ietelacouc(e (N= 437) (N= 414) (N= 2,399) (N= 2,270) United States 
Gender 
Male 34% 35% 63% 63% 43% 
Female 63% 62% 36% 36% 57% 
Missing 3% 3% 1% 1% 0% 
Age 
Under 21 2% 2% 17% 17% 58%% 
22-24 17% 17% 15% 15% 15% 
25-29 26% 25% 20% 20% 10% 
30-39 22% 22% 26% 25% 9% 
40+ 34% 35% 23% 22% 7% 
Missing 0% 0% 0% 0% <1% 


SOURCE: Integrated Postsecondary Education Data System 2018, Random Assignment System, school records. 


NOTE: This exhibit shows the gender and age of the full sample and analytic sample for each experiment, as well as for all postsecondary 
students in the United States. For the full and analytic samples for Experiment 1, data on gender are missing for 14 students. For the full sample 
for Experiment 2, data on age are missing for 6 students. Data on gender are missing for 23 students for both the full and analytic samples in 
Experiment 2. Percentages may not add up to 100 percent due to rounding. 


“ The study evaluated the potential bias of these attrition rates using the U.S. Department of Education’s What Works 
Clearinghouse (WWC) Standards Version 4.0 guidelines for interventions to support postsecondary success. 
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Baseline Equivalence 


Students offered and not offered experimental Pell Grants in the analytic sample had similar 
characteristics at the time of random assignment. Within each experiment, the characteristics of the 
two experimental groups were very similar (Exhibit B.4). In Experiment 2, one of the 18 
characteristics available (percent female) was somewhat different, which is about what would be 
expected by chance. This suggests the random assignment for each experiment created two 
statistically equivalent groups of students, even after sample attrition, such that any differences in 
their later outcomes is the result of being offered an experimental Pell Grant. 
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Exhibit B.4. Characteristics of students offered and not offered experimental Pell Grants at baseline, by experiment 


Characteristic 


(Oitauae 


experimental 
Pell Grant 


Experiment 1 


Nol moyitanee| 
experimental 
Pell Grant 


Difference 


Covariates used to measure impacts (percentages unless otherwise noted) 


Overall 


(icons! 
experimental 
Pell Grant 


Experiment 2 


INolmoyiiauere| 
experimental 
Pell Grant 


IDiiiananas 


Overall 


Age (years) 36.0 35.9 Ll 36.2 31.7 31.4 0.3 31.6 
Female 62.8 70.0 12 64.0 37.7 34.0 3.7* 36.3 
High school graduate 97.8 97.0 0.8 97.3 98.6 98.0 0.6 98.4 
Prior postsecondary experience 97.18 99.0 1.9 96.8 53.6 52.0 1.6 52.5 
Expected family contribution? 1,490.7 1,606.4 -115.6 1,583.9 841.0 1,109.1 -268.1 972.0 
(index) 
Expected family contribution = 0 42.5 43.0 -0.5 42.4 25.4 26.0 -0.6 26.1 
Dislocated worker‘ 22.8 26.0 -3.2 25.4 23.4 22.0 1.4 22.7 
Other characteristics (percentages unless otherwise noted) 
Adjusted gross income ($) 21,956 18,186 3,771 20,670 21,610 23,425 1,814 22,451 
Married 24.7 30.0 -5.3 28.4 21.4 23.0 -1.6 22.1 
Household size (persons) 2.1 2.0 0.1 2.1 2.6 2.7 -0.1 2.6 
Veteran 3.6 2.0 1.6 4.4 4.6 3.0 1.6 4.0 
At risk of homelessness@ 0.0 0.0 0.0 0.0 1.0 1.0 0.0 0.5 
Independent status® 90.6 94.0 -3.4 92.8 84.8 83.0 1.8 84.5 
Average Pell funds disbursed prior to 5,877 6,074 -198 5,597 2,974 2,743 231 2,879 
study period ($) 
Average federal student loans 6,942 8,024 -1,082 7,116 1,054 1,124 -70 1,081 
disbursed prior to study period ($) 
Intended program in high demand 51.6 52.0 -0.4 53.0 57.2 58.0 -0.8 57.5 
Duration of intended program NA NA NA NA 11.4 11.5 -0.1 11.4 
(weeks) 
Enrolled at study school prior to 27.7 24.0 3.7 24.4 14.0 14.0 0.0 13.8 
study period 


SOURCE: Office of Federal Student Aid (FSA), Random Assignment System, school records. 


NOTE: This exhibit contains information for the analysis samples for Experiment 1 (N = 414) and Experiment 2 (N = 2,270). The “Overall” columns (1 and 5) contain the unadjusted means for each student 
characteristic for the analytic sample for each experiment. To test whether the differences reported in the “Difference” columns (4 and 8) are statistically significantly different from zero, the study used 
regression models with each characteristic of interest as the dependent variable, an indicator that the student was offered experimental Pell funds as the independent variable, and school and program 
fixed effects. The difference reported in columns 4 and 8 are the coefficients on the “offered experimental Pell funds” indicator. The exhibit is divided into two panels. The upper panel includes 
information on the characteristics, measured at baseline, that the study included in regression models to estimate impacts of the experiments. The bottom panel includes additional characteristics for 
which the study assessed baseline equivalence (that is, whether students offered Experimental Pell funds and students not offered the funds were similar in terms of characteristics measured at the outset 
of the experiments). 


16 


@ All participants in Experiment 1 were required to have a bachelor’s degree, but a small percentage of these individuals were not reported as having some college in the school data or in FSA data. For every 
record that could be verified after the study had been completed, schools confirmed that this individual did, in fact, have a bachelor’s degree, meaning the reported data had been in error on this indicator. 


>The expected family contribution (EFC), a measure of a family’s financial strength, is an index score calculated based on taxed and untaxed income, assets, and benefits (see https://studentaid.gov/help- 
center/answers/article/what-is-expected-family-contribution). The EFC takes into account whether students are financially dependent on their parents, marital status, and family size, in addition to income, 
assets, and benefits. See formulas here: https://ifap.ed.gov/sites/default/files/attachments/2019-10/2021EFCFormulaGuideOct2019UpdateAttach.pdf. Because the EFC takes these characteristics into 
account, the study included EFC and an indicator for EFC = O as a proxy for these characteristics in regression models to estimate impacts of the experiments (see Exhibit B.7). 


© An individual may qualify as a dislocated worker if they meet one of the following conditions: has lost job, has been laid off or received a lay-off notice, is receiving unemployment benefits due to being 
laid off or losing a job and is unlikely to return to a previous occupation, is self-employed but unemployed due to economic conditions or a natural disaster, is the spouse of an active duty member of the 
Armed Forces and is unemployed or underemployed and is experiencing difficulty in obtaining or upgrading employment, or is a displaced homemaker (a person who previously provided unpaid services 
to the family, is no longer supported by the spouse, is unemployed or underemployed, and is having trouble finding or upgrading employment) (see https://fafsa.gov/fotw2021/help/fahelp69.htm). 


4 A student who is at risk of homelessness is an unaccompanied youth who is homeless or in danger of becoming homeless. Determinations are made on a case-by-case basis (see 
https://fafsa.ed.gov/fotw2021/help/fahelp29a.htm). 
*To be declared an independent, a student must meet at least one of the following criteria: 24 years of age or older, married, a graduate or professional student, a veteran or active duty member of the 


armed forces, an orphan or ward of the court, someone with legal dependents other than a spouse, an emancipated minor, or someone who is homeless or at risk of becoming homeless 
(https://www.thebalance.com/requirements-to-apply-as-a-fafsa-independent-student-4174305). 


* Values differ significantly between the offered experimental Pell and not offered experimental Pell groups, 0.05 level, two-tailed test. 
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B.3. Measures and data sources used in the study 


Outcome measures 


As described in the research questions, the study examined whether expanding Pell Grant eligibility 
improved enrollment in and completion of any postsecondary programs in study schools, as well as 
whether students offered experimental Pell Grants used them. The primary outcomes used to answer 


these research questions are summarized in Exhibit B.5. Study schools provided the data on these 


primary outcomes. 


Exhibit B.5. Primary outcomes 


CO}bixeey rts | 


Research question 1: Did offering experimental Pell Grants increase enrollment in and completion of postsecondary 


SDY=yatevia loyal 


programs in study schools? 


| Data source | 


Timing 


Response rate* 


Enrolled in the Whether the student was School records 12 months after RA 100% 
study school enrolled at a study school (Experiment 1); 
8 months after RA 
(Experiment 2) 
Completed a Whether the student School records 30 months after RA 100% 
program at the completed a program at a (Experiment 1); 
study school study school 10 months after RA 
(Experiment 2) 
Research question 2: What percentage of students who were offered experimental Pell Grants used them? 
Used experimental Proportion of students School records 30 months after RA 100% 
Pell Grant funds offered experimental Pell (Experiment 1); 
(descriptive) Grants that were disbursed 10 months after RA 
experimental Pell Grant (Experiment 2) 
funds 
Experimental Pell Amount of experimental Pell | School records 30 months after RA 100% 
Grant disbursement | Grants disbursed per student (Experiment 1); 
amount 10 months after RA 


(descriptive) 


(Experiment 2) 


NOTE: The follow-up periods for Experiments 1 and 2 are based on the period for which eligible students could use the experimental Pell funds. 
Eligible occupational training programs for Experiment 1 could take up to one year to complete (or up to two years part-time). Eligible FSA- 
approved programs for Experiment 2 were two to four months (8-15 weeks) in duration. 


4 There are no missing data on the primary enrollment and completion outcomes. These outcomes could have a value of 0, indicating that the 


student did not enroll in (complete) a program at the study school, or 1, indicating that the student did enroll in (complete) a program at the study 


school. If a student did not appear in the school records as having enrolled in (completed) a program at the study school, that student would 
receive a value of O for the primary enrollment (completion) outcome. In addition, there are no missing data on the experimental Pell Grant 
disbursement variables. If student records showed that a student who was offered an experimental Pell Grant had no disbursement amount, that 


student received a value of O for the Pell Grant disbursement amount variable, and for the variable indicating whether the student used 
experimental Pell Grant funds. 


RA = random assignment, FSA = Office of Federal Student Aid. 
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The study examined several exploratory outcomes to address the research question (see Exhibit B.6). 
Because the exploratory research questions look beyond the study schools to understand the effects 
of offering experimental Pell Grants, some of these measures were created using multiple data 
sources, described in further detail below. 


Exhibit B.6. Exploratory outcomes 


Outcome | SBY=sivevtucoyet | Data source | Timing | Response rate* 


Exploratory research question 1: Would the effects on enrollment and completion be different if the analysis were 
extended beyond the study schools, or beyond the study’s completion window? 


Enrolled in any school | Whether the student was School records, 12 months after RA 100% 
enrolled at any school NSC (Experiment 1); 
(including the study 8 months after RA 
school) (Experiment 2) 
Completed a program | Whether the student School records, 30 months after RA 100% 
at any school completed a program at NSC (Experiment 1); 
any school (including the 10 months after RA 
study school) (Experiment 2) 
Completed a program | Whether the student School records 30 months after RA 100% 
or still enrolled at the completed a program or (Experiment 1); 
study school was still enrolled at the 10 months after RA 
study school (Experiment 2) 
Completed a program | Whether the student School records, 30 months after RA 100% 
or still enrolled at any | completed a program or NSC (Experiment 1); 
school was still enrolled at any 10 months after RA 
school (including the study (Experiment 2) 
school) 


Exploratory research question 2: Did offering experimental Pell Grants affect the types of programs that students 
enrolled in or completed? 


Enrolled in a high- Whether the student was School records, 12 months after RA 98% 
demand program at enrolled in a program DOL (Experiment 1); 
the study school associated with a high- 8 months after RA 

demand occupation in the (Experiment 2) 

state it was offered 
Completed a high- Whether the student School records, 30 months after RA 99% 
demand program at completed a program DOL (Experiment 1); 
the study school associated with a high- 10 months after RA 

demand occupation in the (Experiment 2) 

state it was offered 
Completed a Whether the student School records 30 months after RA 93% 
certificate program at | completed a certificate (Experiment 1); 
the study school program at the study 10 months after RA 

school (Experiment 2) 
Enrolled ina Whether the student was School records, 12 months after RA 100% 
certificate program at | enrolled ina certificate NSC (Experiment 1); 
any school? program at any school 8 months after RA 

(including the study (Experiment 2) 

school) 
Completed a Whether the student School records, 30 months after RA 100% 
certificate program at | completed a certificate NSC (Experiment 1); 
any school program at any school 10 months after RA 

(including the study (Experiment 2) 

school) 
Completed an Whether the student School records 30 months after RA 93% 
associate’s degree completed an associate’s (Experiment 1); 
program at the study degree program at the 10 months after RA 
school study school (Experiment 2) 
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Outcome Definition Data source Timing Response rate? 


Enrolled in an Whether the student was School records, 12 months after RA 100% 
associate’s degree enrolled in an associate’s NSC (Experiment 1); 
program at any degree program at any 8 months after RA 
school? school (including the study (Experiment 2) 
school) 
Completed an Whether the student School records, 30 months after RA 100% 
associate’s degree completed an associate’s NSC (Experiment 1); 
program at any school | degree program at any 10 months after RA 
school (including the study (Experiment 2) 
school) 


Exploratory research question 3: Did offering experimental Pell Grants impact students’ use of federal student loans? 


Federal student loan Whether the student took FSA 30 months after RA 100% 
take-up out federal student loans (Experiment 1); 

(that is, whether the 10 months after RA 

student hada positive (Experiment 2) 


dollar amount of federal 
student loans disbursed) 


Federal student loans | Amount of federal student | FSA 30 months after RA 100% 
disbursed at any loans disbursed per (Experiment 1); 
school (descriptive) student 10 months after RA 


(Experiment 2) 
NOTE: The follow-up periods for Experiments 1 and 2 are based on the period for which eligible students could use the experimental Pell funds. 
Eligible occupational training programs for Experiment 1 could take up to one year to complete (or up to two years part-time). Eligible programs 
approved by FSA for Experiment 2 were two to four months (8-15 weeks) in duration. 


@ There are no missing data on the enrollment and completion outcomes at any school. These outcomes could have a value of 0, indicating that the 
student did not enroll in (complete) a program at any school, or 1, indicating that the student did enroll in (complete) a program at any school. If a 
student did not appear in the school records or in the NSC as having enrolled in (completed) a program, that student would receive a value of O for 
the exploratory enrollment (completion) outcome at any school. There are missing data on whether the program a student completed was in high 
demand; the study was unable to determine whether some programs were in high demand due to missing Classification of Industrial Program 
codes. There are also missing data on the type of degree or credential associated with programs enrolled in or completed at study schools—some 
schools did not provide this information for all students in the study sample. Finally, there are no missing data on the federal loan disbursement 
variables. If FSA records showed that a student had no federal student loan disbursement amount, that student received a value of O for the 
disbursement amount variable, and for the variable indicating whether the student used federal loans. 


> Due to a large amount of missing data in school records, the study is not able to examine impacts on enrollment in different credential or degree 
programs at study schools. 


NSC = National Student Clearinghouse, RA = random assignment, FSA = Office of Federal Student Aid, DOL = U.S. Department of Labor. 


Two of the exploratory outcomes—enrollment in and completion of a high-demand program ina 
study school—required several steps to create. To determine whether a program was in a high- 
demand occupation in its state, the study adapted criteria established by the U.S. Department of 
Labor’s Occupational Information Network (O*NET) for “Bright Outlook” occupations that were 
expected to grow rapidly or have large numbers of job openings between 2014 and 2016, midway 
through the study period.* High-demand occupations needed to meet at least one of the following 
three criteria: 


1. Projected to grow rapidly. These occupations were projected to grow much faster than average 
(employment increase greater than one standard deviation from the mean growth rate in that 
state) over the period 2014-2016. 


2. Projected to have large numbers of openings. These occupations were projected to have a high 
number of openings relative to other occupations in that state (occupations with an average 


5 See https://www.onetonline.org/find/bright. 
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annual number of openings that is greater than one standard deviation from the mean of average 
annual openings for all occupations in that state). 


3. New and emerging occupations. New workforce requirements, including changes in technology, 
society, law or business practices, constantly lead to new and emerging occupations in the 
United States. Such occupations were identified within high-growth industries, as described in 
the New and Emerging (N&E) Occupations Methodology Development Report.® 


The process for determining whether a program was associated with a high-demand occupation 
involved three steps: 


1. The study associated programs with specific occupations using crosswalks from the Department’s 
National Center for Education Statistics (NCES)." 


2. The study obtained labor market information on those occupations in the state in which the 
associated program was offered from the Projections Central State Occupational Projections 
database. 


3. Labor market information for each occupation was analyzed against O*NET Bright Occupations 
criteria. For a given program, if the associated occupations met any of the three criteria within 
the state where the program was located, then the program was determined to be in a high- 
demand occupation. 


Background characteristics 


In addition to measuring the primary and exploratory outcomes, the study examined background 
characteristics of students to understand and account for any differences between students offered 
experimental Pell Grants and those who did not receive the offer, as well understand the 
characteristics of the students who used the experimental Pell Grant funds they were offered. The 
student characteristics included in the analysis to estimate impacts are summarized in Exhibit B.7. 


Exhibit B.7. Background characteristics included in the analysis 


Nyattelsets Data Response 
characteristic Definition source Rationale rate* 
Age Students’ age at time of random RAS Increased access to financial aid 100% 
assignment (calculated as date of has had a significant effect on the 
assignment minus date of birth/365.25) enrollment of older, nontraditional 
students? 
Gender Binary variable with 1 indicating the FSA, Although the majority of 99% 
participant is female and 0 indicating school postsecondary students are 
male records female, enrollment of both males 
and females has dropped in recent 
years, with a slightly larger drop 
for female students‘ 


16 https://www.onetcenter.org/reports/NewEmerging.html. 


'’ https://nces.ed.gov/ipeds/cipcode/resources.aspx?y=55. 


8 http://www.projectionscentral.com/Projections/ShortTerm. 


'° There were no data for Washington for this period, so the study obtained those data directly from the state of Washington. 
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Nyatteloels Data Response 


characteristic Definition source Rationale rate@ 
High school Binary variable with 1 indicating the FSA, Research shows a relationship 98% 
diploma or participant holds a high school school between completing high school 
equivalent diploma or GED/state certificate at the | records and postsecondary enrollment and 

time of FAFSA completion and 0 completion? 

indicating the participant does not 
Prior Binary variable with 1 indicating that FSA, Only applies to Experiment 2 98% 
postsecondary the study participant had some college | school because all Experiment 1 
experience experience prior to participating in the | records participants already had a 
(Experiment 2 experiment bachelor’s degree 
only) 
Expected family Continuous variable denoting the FSA To encompass income, assets, 89% 
contribution student’s or the student’s family’s benefits, and family size, which 
(EFC)° ability to pay the cost of school affect students’ need for financial 

attendance aid 
Indicator for EFC | Indicator equals 1ifEFC=0,O0ifEFC> | FSA To capture nonlinearities at EFC = 89% 
=0 0 0 (as students with an EFC = 0 may 


respond differently to the offer of 
experimental Pell Grants) 


Dislocated Binary variable with 1 indicating the FSA To capture employment status of 89% 
worker‘ participant is a dislocated worker and study participants 

O indicating the participant is not a 

dislocated worker 


There are no missing data for age; the Random Assignment System (RAS) contains the date of birth for each student in the study sample. The study 
obtained data on gender, high school diploma or equivalent, and prior postsecondary experience from FSA and school records. For students who 
were missing information from FSA on these variables, the study used school records. 

> Seftor, N.S., and Turner, S. (2002). Back to School: Federal Student Aid Policy and Adult College Enrollment. Journal of Human Resources, 37(2), 
pp. 336-352. 


© See https://nces.ed.gov/fastfacts/display.asp?id=98. 


4 See, for example, Engberg, M. E., & Wolniak, G. C. (2010). Examining the effects of high school contexts on postsecondary enrollment. Research in 
Higher Education, 51(2), 132-153. 


© The expected family contribution, a measure of a family’s financial strength, is an index score calculated based on taxed and untaxed income, 
assets, and benefits (see https://studentaid.gov/help-center/answers/article/what-is-expected-family-contribution). 


‘An individual may qualify as a dislocated worker if they meet one of the following conditions: has lost job, has been laid off or received a lay-off 
notice, is receiving unemployment benefits due to being laid off or losing a job and is unlikely to return to a previous occupation, is self-employed 
but unemployed due to economic conditions or a natural disaster, is the spouse of an active duty member of the Armed Forces and is unemployed 
or underemployed and is experiencing difficulty in obtaining or upgrading employment, or is a displaced homemaker (a person who previously 
provided unpaid services to the family, is no longer supported by the spouse, is unemployed or underemployed, and is having trouble finding or 
upgrading employment) (see https://fafsa.gov/fotw2021/help/fahelp69.htm). 


when available. The study obtained information on expected family contribution and dislocated worker status from FSA. FSA records were 
available for 89 percent of the study sample. 


RAS = Random Assignment System, FSA = Office of Federal Student Aid, GED = General Equivalency Diploma. 
Data sources 


The study obtained data from the RAS used for the lottery, student records from study schools, as 
well as administrative data from the Office of Federal Student Aid (FSA) and other sources to create 
the variables and outcome measures used in analyses (Exhibit B.8). To reduce the burden on study 
schools, the study asked that they provide a limited number of data items in the RAS and either 
extract student records in whatever format available, or provide student data in an excel spreadsheet 
with pre-specified fields. Data entered into the RAS primarily served to identify the study participants 
and keep track of which experiment and group they were assigned to; schools also entered students’ 
dates of birth into the RAS. Study schools then provided records on students’ program enrollment, 
completion, and experimental Pell Grant receipt, as well as information on students’ high school 
graduation status and prior postsecondary experience. FSA provided data from student Free 
Applications for Federal Student Aid (FAFSAs) on high school graduation status, prior postsecondary 
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experience, expected family contribution, and dislocated worker status. FSA also provided 
information on federal student aid received. Data from the National Student Clearinghouse were 
used to explore enrollment and completion outside of the study schools.”° Finally, the characteristics 
of schools and programs in the study came from the Integrated Postsecondary Education Data 
System and the Department of Labor. 


Exhibit B.8. Data sources 


Source Data item AWdotcumetle-Mololedietarer Use in study 
Random Assignment e Experiment type (Experiment1 | Fall 2012- e Define the study sample 
System or Experiment 2) Spring 2017 e Facilitate obtaining 
e Random assignment status sociodemographic information 
(offered or not offered an and college enrollment and 
experimental Pell Grant) financial aid data for study 
e Program of interest participants from other 
¢ Social security number source? 
° Date of birth e Create covariate (age) for the 
analysis 
e Define subgroup (under/over 
25) 
Study schools’ e Program enrollment/exit dates Winter 2015- e Measure academic outcomes 
student records ° Program of study Summer 2018 for study participants at study 
schools 


e Program completion 


e Credential attainment e Measure Seek i. 

¢ Experimental Pell Grant receipt SADE HEHE Vet ranks 
e Create covariates for the 

e Gender 


analysis 
e Define subgroups 
e Describe programs attended 


e High school completion status 
e Prior postsecondary experience 


Federal Student Aid e High school completion status Winter 2020 e Measure financial aid 
databases e Prior postsecondary experience outcomes for study 
e Expected family contribution participants 
¢ Dislocated worker status * Create covariates for the 
analysis 


e © Marital status 

e Veteran status 

e Atrisk of homelessness status 
e Federal financial aid receipt 


e Define subgroups 


e Describe other background 
characteristics for the study 


sample 
National Student e Program enrollment at any Spring 2018 e Measure postsecondary 
Clearinghouse school outcomes at schools 
e Program completion at any participants may have 
school attended (in addition to study 
schools) 


e Credential attainment at any 
school 


20 NSC aims to obtain data from all U.S. postsecondary schools and, in practice, receives it from most. As of 2017, the schools that 
reported data to the NSC enrolled approximately 97 percent of all U.S. postsecondary students and 100 percent of the students in 
public, two-year schools (which represent almost three-fourths of the schools in the study). However, some schools do not 
provide data to NSC. Private, for-profit schools in particular are less likely to report to NSC (approximately 25 percent of students 
enrolled in these types of two-year schools are reported to NSC), and several of the study schools fall into this category as well. 
Overall, study schools associated with approximately 70 percent of the study sample reported to NSC. Thus, the study only used 
NSC data to supplement school records and explore enrollment and completion in any postsecondary institution, which was 
defined as whether the student was enrolled or completed a program either according to school records or NSC records. 
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Source Data item AY date metcte-Mololecdiataret Use in study 


Integrated e Type of school (public, private, | Spring 2018 e Compare the characteristics of 
Postsecondary for-profit, not-for-profit) study schools to those of 
Education Data e Size of school postsecondary schools more 
System e Graduation rate broadly 
e Number/percentage of students 
receiving financial aid 


e Student age 
e Student gender 


Department of Labor | e High-growth occupations Spring 2018 e Assess the extent to which the 
databases e Industries “in-demand” approved programs and 
programs in which eligible 
students expressed interest 
were in high-demand 
occupations within their 
respective states 


Minimum detectable effects 


Studies vary in how reliably they can measure effects of different sizes. Like political polls, studies 
with larger samples—here, number of eligible students interested in experimental Pell Grants in the 
analytic sample - are better able to measure smaller differences with certainty than are studies with 
smaller samples. The “minimum detectable effect” or MDE can be calculated for each outcome to 
identify the smallest effect the study could reliably detect given its sample size and design. These 
calculations suggest that the study could reliably detect moderate to large effects of the offer of 
experimental Pell Grants on enrollment and completion (see Exhibit B.9). For example, for 
Experiment 1, the minimum detectable effect is 10.7 percentage points for enrollment and 10.3 
percentage points for completion. Because the sample for Experiment 2 is larger, the minimum 
detectable effects are smaller: five percentage points for enrollment and 5.1 percentage points for 
completion. 


Exhibit B.9. Minimum detectable effects for primary outcomes, by experiment 


Experiment 1 - Experiment 1 - Experiment 2 - Experiment 2 - 


Enrollment Completion of a Enrollment in a OXoyss}o) (in Coyo Moya) 
in a study school program at a study study school program at a study 


ye store) | Ye store) | 


All participants | 44 | 107 | 44 | 103 | 2270 | 50 | 2270 | 5.1 


Source: Random Assignment System, school records, Office of Federal Student Aid. 


Note: These calculations were based on the actual sample sizes and data collected for the study. In calculating MDEs, the study used an 80 percent 
power level and a 5 percent level of statistical significance. 


MDE = minimum detectable effect. 
B.4. Analytic approach 


Examining the effects of the experiments 


To examine the effect of each pilot expansion to Pell Grant eligibility—allowing students with a 
bachelor’s degree to receive experimental Pell funds in Experiment 1 and allowing students who 
enrolled in very short-term occupational training programs to receive experimental Pell funds in 
Experiment 2—the study estimated the effects separately for each experiment using the following 
ordinary least squares regression model: 
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(1) Yj, = BT, +Xjy +a, +6, + Eig: 
where the terms are as follows: 


e Y,. is the outcome of individual i in school s (that is, the school in which random assignment 


occurred); 


e _ T, isa treatment indicator that equals one if the participant was randomly assigned to receive an 


offer of experimental Pell funds and zero otherwise; 


o X j is a vector of baseline individual characteristics (Exhibit B.7); 


°  @, isa vector of fixed effects for each school and 6 . is a vector of fixed effects for each program 


group, based on the program in which study participants expressed interest at the time of 
random assignment (each program group within a school is a separate stratum);! 


© Ey is an individual-level random error term; and 


e  # and y are vectors of parameters to be estimated. 


The model above was used to estimate the effects of offering experimental Pell Grants on each 
primary and exploratory outcome among the analytic sample described in section B.2.”” Because the 
model estimates the effect of the offer (rather than the actual use) of experimental Pell Grants, the 
results are intent-to-treat impacts. Understanding the effects of making Pell Grants available was the 
most policy relevant question for this study, as it would provide evidence on what might happen if 
the eligibility waivers tested in the study were made official policy. 


The model included the student characteristics listed in Exhibit B.7, measured at or before the time 
of random assignment, as covariates to control for potential differences between those offered and 
those not offered experimental Pell Grants that could influence outcomes; for example, as shown in 
Exhibit B.4, these groups in Experiment 2 were significantly different in terms of the percentage of 
female students. Including student characteristics associated with the outcomes can also improve the 
precision of the estimated effects. 


Study participants with missing covariate data were included in the analyses. For each missing 
covariate, analyses replaced missing values with a placeholder value (zero) and included an indicator 
for whether the participant originally had a missing value for that covariate in the regression model. 
Simulations suggest that this approach leads to limited bias under the conditions that are likely to 


2! Program groups were based on Standard Occupational Classification codes, and in some cases, multiple SOC codes were 
grouped together to reflect similar types of programs. Some participating schools (19 out of 46) offer only one program. For these 
schools, the school-level control variables are collinear with the program group indicators. 


2 A multiple-comparisons adjustment was not made to the significance of the estimates because the two primary outcomes— 
enrollment and program completion at participating schools—are in different outcome domains. 
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apply in this study.** Appendix D describes the results of a sensitivity analysis that used a different 
method—casewise deletion—to account for missing data. 


Examining the effects of the experiments for subgroups 


For each experiment the study investigated the effect of offering experimental Pell Grants on the two 
primary outcomes within subgroups defined by student, program, and local characteristics to better 
understand how different types of students might respond if the eligibility waivers tested in the 
experiments were made official policy. Exhibit B.10 defines and provides the rationale for examining 
each subgroup. 


Exhibit B.10. Subgroups examined 


Subgroup Rationale 


Gender (male/female) 


Although the majority of postsecondary students are female, enrollment of both males 
and females has dropped in recent years, with a slightly larger drop for female students.* 


Age (under 25/25 or older) 


Increased access to financial aid has had a significant effect on the enrollment of older, 
nontraditional students.” 


Expected family contribution 
(EFC = O, EFC > O¢ 


Individuals with very high financial needs may respond differently to having access to an 
experimental Pell Grant. 


Dislocated worker/not a 
dislocated worker® 


Individuals who have been dislocated from their previous employment may respond 
differently to having access to an experimental Pell grant. 


Enrolled in study school prior to 
random assignment/not enrolled 
in study school prior to random 
assignment 


Some study participants were already enrolled in a program at the time of random 
assignment, so the study examined whether these students might respond differently to 
having access to an experimental Pell Grant. 


In a high-demand occupation/not 
in a high-demand occupation*® 


Occupational programs eligible for experimental Pell Grant funds were intended to meet 
local or regional workforce needs. Therefore, the study examined whether effects 
differed for students who expressed interest in a high-demand program at the time of 
random assignment. 


Annual unemployment rate 
(county rate above/below national 
average) 


Because enrollment in community colleges has responded to local economic conditions 
such as unemployment rates,’ The study examined whether effects differ based on the 
unemployment rate in the county in which the study school was located. 


4 See https://nces.ed.gov/fastfacts/display.asp?id=98. 
> Seftor, N.S., and Turner, S. (2002). Back to School: Federal Student Aid Policy and Adult College Enrollment. Journal of Human Resources, 37(2), 
pp. 336-352. 


© The expected family contribution, a measure of a family’s financial strength, is calculated based on taxed and untaxed income, assets, and benefits 
(see https://studentaid.gov/help-center/answers/article/what-is-expected-family-contribution). 


4 An individual may qualify as a dislocated worker if they meet one of the following conditions: has lost job, has been laid off or received a lay-off 
notice, is receiving unemployment benefits due to being laid off or losing a job and is unlikely to return to a previous occupation, is self-employed 
but unemployed due to economic conditions or a natural disaster, is the spouse of an active duty member of the Armed Forces and is unemployed 
or underemployed and is experiencing difficulty in obtaining or upgrading employment, or is a displaced homemaker (a person who previously 
provided unpaid services to the family, is no longer supported by the spouse, is unemployed or underemployed, and is having trouble finding or 
upgrading employment) (see https://fafsa.gov/fotw2021/help/fahelp69.htm). 


*Section B.3 describes how the study determined whether a program was in high demand. 


Hillman, N. W., & Orians, E. L. (2013). Community colleges and labor market conditions: How does enrollment demand change relative to local 
unemployment rates?. Research in Higher Education, 54(7), 765-780. 


23 Puma, M. J., Olsen, R. B., Bell, S. H., & Price, C. (2009). What to Do when Data Are Missing in Group Randomized Controlled 
Trials. NCEE 2009-0049. National Center for Education Evaluation and Regional Assistance. 
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For each subgroup of interest, the study estimated the effects of offering experimental Pell Grants 
under each of the two experiments using the following regression model, which adds an interaction 
term to the benchmark model in Equation (1): 


@ Vg =BT,+B)(T*Wyg)+Xiy +0, +5, +65. 


Here, W represents the relevant subgroup variable (for example, an indicator for whether the 
participant is female). All subgroup variables of interest are included in the vector of baseline 
individual characteristics X , and therefore Equation (2) also estimates the main effect of W . When 


W isa binary indicator, the coefficient on the interaction term B, represents how the impact for 
members of that subgroup differs from the impact for others, captured by ,. Thus, the impact for 


the subgroup is equal to B, + By. All other terms are as defined in Equation (1). 


Examining student characteristics correlated with use of experimental Pell Grants 


To understand whether certain types of students who were offered experimental Pell Grants were 
more likely to go on to take up the offer—for example, whether these students were more likely to be 
female, dislocated workers, and so on—the study conducted a supplemental regression analysis 
comparing the key student characteristics described in Exhibit B.7 between students who received 
experimental Pell Grant funds and those who did not. First, the study created an indicator to denote 
whether a student received a positive dollar amount of experimental Pell Grant funds. The study 


then regressed this indicator on the vector of baseline individual characteristics X ;» as defined in 


Equations (1) and (2), and examined the coefficient on each characteristic variable, given by the 
vector 7 as defined in Equations (1) and (2). 
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APPENDIX C. SUPPORTING DETAILS FOR KEY FINDINGS 


Whereas Appendix B describes how study analyses were conducted, this appendix provides 
additional information on the results of the analyses. It first presents details supporting the main and 
exploratory findings shown in the report, including the effects of the experiments on key subgroups 
of students and schools. The appendix then describes the different types of programs that students in 
each experiment expressed interested in, enrolled in, and completed, which are only briefly noted in 
the body of the report. 


C.1. Additional information supporting estimates of the experiments’ effects 


As described in the report, the study found large positive and statistically significant impacts on 
enrollment and completion at study schools for both experimental expansions to Pell Grant 
eligibility. To help readers better understand the results for each experiment, this section presents 
the findings for exploratory outcomes (see Exhibit B.6 in Appendix B). 


The exploratory analyses for Experiment 1, summarized in Exhibit C.1, suggests the following: 


e The findings were consistent when examining enrollment and completion at any school or 
examining continued enrollment, in addition to completion. Because students not offered 
experimental Pell Grants at the study schools might be more likely than those offered grants to 
participate in occupational programs elsewhere, it was important to examine impacts on 
enrollment in and completion of a program at any school. The impacts on these outcomes were 
very similar in both magnitude and statistical significance to the impacts on enrollment and 
completion measured at study schools. Students not offered experimental Pell funds may also 
have opted for more traditional programs, which typically take longer to complete than the 
study’s data collection period was able to capture. To address this, the study explored the 
impacts on whether students completed or were still enrolled at the end of the study’s follow-up 
window; this impact was also similar to the primary completion impact reported previously. 
Together these results suggest that, compared to students offered experimental Pell Grants, 
those not offered grants did not choose to pursue longer programs or programs at other schools 
at rates high enough to affect the study’s main findings. 


e Offering experimental Pell Grants led to higher rates of completion of high-demand programs 
and programs yielding certificates. Students who were offered experimental Pell Grants were 
about 11 percentage points more likely to complete a high-demand program at study schools, 
compared with students not offered experimental Pell funds. Moreover, students who were 
offered experimental Pell Grants were 19 percentage points more likely to complete a certificate 
program at study schools or at any school. Few students completed associate’s degree programs 
at study schools or any school; however, the percentage of students offered experimental Pell 
Grants who completed an associate’s degree program at any school was slightly lower than the 
percentage of students not offered experimental Pell Grants who completed such a program. 
Because this difference is small and not statistically significant at the conventional five percent 
level, this finding does not provide convincing evidence that the offer of experimental Pell funds 
reduced the rate of associate’s degree completion. 
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e Offering experimental Pell Grants had no impact on students’ use of federal student loans. The 
share of students who were offered experimental Pell funds and used federal student loans was 
similar to the share of students not offered experimental Pell funds who used federal loans. The 
federal student loan amounts disbursed during the study period were also similar between these 
two groups. 


Exhibit C.1. Primary and exploratory impact estimates for Experiment 1 


Oicocael Not offered 
experimental experimental 
Pell Grant Pell Grant 
Standard 

Outcome (percentage unless otherwise noted) (uxey 
Primary 
Enrolled in the study school 77.9 254 51.9 160 26.0 4.0 0.000 
Completed a program at the study school 52.4 254 35.6 160 16.7 4.3 0.000 


Exploratory research question 1: Would impacts on enrollment and completion be different if the analysis were 
extended beyond the study schools offering experimental Pell funds, or beyond the study’s completion window? 


Enrolled in any school 82.7 254 63.1 160 19.6 3.9 0.000 
Completed a program at any school 54.2 254 39.4 160 14.9 4.5 0.001 
Completed a program or still enrolled at the study 54.7 254 38.8 160 15.9 4.3 0.000 
school 

Completed a program or still enrolled at any school 64.2 254 48.1 160 16.1 4.5 0.000 


Exploratory research question 2: Did offering experimental Pell Grants affect the types of programs students enrolled in 
or completed? 


Enrolled in a high-demand program at the study 57.0 229 38.2 144 18.8 4.2 0.000 
school 

Completed a high-demand program at the study 40.1 248 29.5 156 10.6 3.9 0.006 
school* 

Completed a certificate program at the study school 37.2 254 18.1 160 19.1 4.0 0.000 
Enrolled in a certificate program at any school 48.1 254 31.3 160 16.8 4.1 0.000 
Completed a certificate program at any school 39.1 254 20.0 160 19.1 4.2 0.000 
Completed an associate’s degree program at the 1.0 254 2.5 160 -1.5 1.2 0.199 
study school 

Enrolled in an associate’s degree program at any 4.8 254 5.6 160 -0.9 2.3 0.699 
school 

Completed an associate’s degree program at any 3.3 254 6.3 160 -3.0 1.7 0.089 
school 

Exploratory research question 3: Did offering experimental Pell Grants impact students’ use of federal student loans? 
Federal student loan use 33.2 254 36.9 160 3.7 4.3 0.390 
Federal student loans disbursed at any school ($)° 4,689 254 4,214 160 475 1,193 0.691 


SOURCE: Random Assignment System, school records, Office of Federal Student Aid, National Student Clearinghouse. 


NOTE: Impact estimates were calculated using the approach described in Section B.4 of Appendix B. Exhibit B.6 provides details on the outcomes 
reported in this exhibit. Column 5 in this appendix contains impact estimates that are regression-adjusted for student socioeconomic characteristics 
measured before random assignment, as described in Appendix B. The “Offered experimental Pell Grant” group mean (column 1) equals the sum of 
the unadjusted control group mean (column 3) and the regression-adjusted impact estimate (column 5). Column 2 contains the number of students 
in the “Offered experimental Pell Grant” group who were included in the analysis, and column 4 contains the number of students in the “Not 
offered experimental Pell Grant” group. Column 6 reports the standard error of the impact in column 5, and column 7 reports the probability that 
the impact in column 5 could have been observed by chance, with small p-values indicating a low likelihood that the impact is due to chance and 
not to the offer of experimental Pell funds. Percentages may not add up due to rounding. 

4To determine whether a program was in a high-demand occupation in its state, the study adapted criteria established by the U.S. Department of 
Labor’s Occupational Information Network (O*NET) for “Bright Outlook” occupations that were expected to grow rapidly or have large numbers of 
job openings between 2014 and 2016, midway through the study period. Additional details on how the study determined whether a program was in 
high demand are in Section B.3 of Appendix B. 

> The federal student loan disbursement amounts reported in this table are averages taken over students who did and did not take out student loans 
(that is, the averages include zeroes). Among the subset of students with positive federal student loan disbursement amounts, the average amount 
disbursed was $12,335 for Experiment 1 overall: $12,950 for those offered experimental Pell funds, and $11,429 for those not offered experimental 
Pell funds. 
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Exploratory findings for Experiment 2 largely mirrored those for Experiment 1. For Experiment 2, 
Exhibit C.2 suggests the following: 


e The findings were consistent when examining enrollment and completion at any school or 
examining continued enrollment, in addition to completion. As with Experiment 1, impacts on 
enrollment in and completion of a program at any school were examined because students not 
offered experimental Pell Grants might have been more likely than those offered grants at the 


study schools to participate in occupational programs elsewhere. The impacts on these outcomes 
were very similar to the impacts on enrollment and completion measured at study schools. These 


results suggest that, compared to students offered experimental Pell Grants, those not offered 
grants did not choose to pursue programs at other schools at rates high enough to affect the 
study’s main findings. Students not offered experimental Pell funds may also have opted for 


traditional programs already eligible for Pell grants, which typically take longer to complete than 


the study’s data collection period was able to capture. To address this, the study explored the 


impacts on whether students completed or were still enrolled at the end of the study’s follow-up 


window; this impact was also similar to the primary completion impact. Together these findings 
suggest that, compared to students offered experimental Pell Grants, those not offered the grants 


did not pursue longer programs or programs at other schools at rates high enough to affect the 


study’s main findings. 


e Offering experimental Pell Grants led to higher rates of completion of FSA-approved and high- 


demand programs, as well as programs yielding certificates. Students who were offered 


experimental Pell Grants were far more likely than those not offered the grants to complete an 


FSA-approved program at study schools, with an impact of almost 40 percentage points. In 


addition, students offered experimental Pell Grants were about 8 percentage points more likely 


to complete a high-demand program at study schools and about 6 percentage points more likely 


to complete a certificate program. As with Experiment 1, though few students completed 


associate’s degree programs overall, the percentage of students offered experimental Pell Grants 


who completed an associate’s degree program at any school was slightly lower than the 
percentage of students not offered experimental Pell who completed such a program. Because 


this difference is small and not statistically significant at the conventional five percent level, this 


finding does not provide convincing evidence that the offer of experimental Pell funds reduced 


the rate of associate degree completion. 


e Offering experimental Pell Grants had no impact on students’ use of federal student loans. As 
with Experiment 1, the share of students offered experimental Pell funds who used federal 


student loans was similar to the share of students not offered experimental Pell funds who used 


federal loans. The federal student loan amounts disbursed during the study period were also 
similar between these two groups. 
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Exhibit C.2. Primary and exploratory impact estimates for Experiment 2 


(itoyset Not offered 
experimental experimental 
Pell Grant Pell Grant 
Standard 
Outcome (percentage unless otherwise noted) uxey 
Primary 
Enrolled in the study school 66.4 1,363 51.8 907 14.6 1.8 0.000 
Completed a program at the study school 47.0 1,363 37.7 907 9.3 1.9 0.000 
Exploratory research question 1: Would impacts on enrollment and completion be different if the analysis were 
extended beyond the study schools offering experimental Pell funds, or beyond the study’s completion window? 
Enrolled in any school 78.2 1,363 64.2 907 14.1 1.8 0.000 
Completed a program at any school 64.3 1,363 54.9 907 9.4 2.0 0.000 
Completed a program or still enrolled at the study 54.6 1,363 44.5 907 10.1 1.9 0.000 
school 
Completed a program or still enrolled at any 66.3 1,363 57.1 907 9.2 2.0 0.000 
school 
Exploratory research question 3: Did offering experimental Pell Grants effect the types of programs students enrolled in 
or completed? 
Enrolled in an FSA-approved program at the 59.9 1,363 9.7 907 50.2 1.6 0.000 
study school 
Completed an FSA-approved program at the 44.3 1,363 5.4 907 38.9 1.7 0.000 
study school 
Enrolled in a high-demand program at the study 58.2 1,344 45.2 903 13.1 1.7 0.000 
school 
Completed a high-demand program at the study 40.9 1,354 32.8 905 8.1 1.8 0.000 
school" 
Completed a certificate program at the study 39.6 1,224 33.4 848 6.3 1.9 0.001 
school 
Enrolled in a certificate program at any school 56.5 1,363 48.3 907 8.2 1.7 0.000 
Completed a certificate program at any school 44.1 1,363 38.0 907 6.1 1.9 0.001 
Completed an associate’s degree program at the 0.0 1,224 0.0 848 0.0 NA NA 
study school 
Enrolled in an associate’s degree program at any 3.3 1,363 2.7 907 0.6 0.7 0.365 
school ee on 
Completed an associate’s degree program at any 0.1 1,363 0.6 907 -0.4 0.2 0.065 
school 
Exploratory research question 5: Did offering experimental Pell Grants impact students’ use of federal student loans? 
Federal student loan use 91 1,363 8.3 907 0.9 1.1 0.452 
Federal student loans disbursed at any school ($)° 391 1,363 393 907 1.98 74.46 0.979 


Source: Random Assignment System, school records, Office of Federal Student Aid, National Student Clearinghouse. 


Note: Impact estimates were calculated using the approach described in Section B.4 of Appendix B. Exhibit B.6 provides details on the outcomes 
reported in this exhibit. Column 5 contains impact estimates that are regression-adjusted for student socioeconomic characteristics measured 
before random assignment, as described in Appendix B. The “Offered experimental Pell Grant” group mean (column 1) equals the sum of the 
unadjusted control group mean (column 3) and the regression-adjusted impact estimate (column 5). Column 2 contains the number of students in 
the “Offered experimental Pell Grant” group who were included in the analysis, and column 4 contains the number of students in the “Not offered 
experimental Pell Grant” group. Column 6 reports the standard error of the impact in column 5, and column 7 reports the probability that the 
impact in column 5 could have been observed by chance, with small p-values indicating a low likelihood that the impact is due to chance and not to 
the offer of experimental Pell funds. Percentages may not add up due to rounding. 

@To determine whether a program was in a high-demand occupation in its state, the study adapted criteria established by the U.S. Department of 
Labor’s Occupational Information Network (O*NET) for “Bright Outlook” occupations that were expected to grow rapidly or have large numbers of 
job openings between 2014 and 2016, midway through the study period. Additional details on how the study determined whether a program was in 
high demand are in Section B.3 of Appendix B. 

> The federal student loan disbursement amounts reported in this table are averages taken over students who did and did not take out student loans 
(that is, the averages include zeroes). Examining the subset of students with positive federal student loan disbursement amounts, the average 
amount disbursed was $4,296 overall: $4,021 for those offered experimental Pell funds, and $4,757 for those not offered experimental Pell funds. 


FSA = Federal Student Aid Office. 
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C.2. Additional information supporting estimates of the experiments’ effects on subgroups of 
students, schools, and communities 


The study examined whether impacts on the primary outcomes (enrollment in and completion of 
any program at study schools) varied for subgroups defined by student, program, and local 
characteristics (see Exhibit B.10 for a description and rationale of the study’s choice of subgroups). 
This section reports estimates of the impacts of the offer of experimental Pell Grants on the primary 
outcomes by subgroup (for example, males and females); it also reports the difference in impacts 
across subgroups (Exhibits C.3 and C.4). 


Exhibit C.3. Impacts on enrollment in and completion of a program at study schools for 
Experiment 1, by subgroup 


Completed a program at the 


Enrolled in the study school study school 


Subgroup Impact Impact 


Student characteristic 


Male 256 22.8 0.000 14.0 0.011 
Female 144 33.2 0.000 22.5 0.005 
Difference -10.4 0.255 -8.5 0.389 
Under 25 77 18.8 0.010 5.3 0.496 
Over 25 337 27.7 0.000 19.5 0.000 
Difference 8.9 0.232 -14.2 0.075 
EFC = 0? 142 21.0 0.002 15.5 0.037 
EFC > O 193 22.3 0.000 7.2 0.271 
Difference -1.2 0.894 8.3 0.403 
Dislocated worker? 85 46.1 0.000 28.4 0.004 
Not a dislocated worker 250 14.0 0.005 5.3 0.340 
Difference 32.0 0.002 23.1 0.040 
Enrolled prior to random 101 25.8 0.000 4.5 0.559 
assignment 
Not enrolled prior to random 313 26.1 0.000 20.7 0.000 
assignment 
Difference -0.2 0.976 -16.2 0.060 
Program characteristic 
High-demand occupation‘ 213 19.9 0.000 10.0 0.063 
Not a high-demand occupation 189 33.7 0.000 25.8 0.000 
Difference -13.8 0.027 -15.8 0.019 
Local characteristic 
County unemployment rate 320 28.1 0.000 17.6 0.000 
_abovenationalaverage | 
County unemployment rate 94 18.2 0.019 13.7 0.100 
below national average 
Difference 10.0 0.234 3.9 0.666 


SOURCE: Random Assignment System, school records, Office of Federal Student Aid. 


NOTE: This exhibit reports the number of students in each subgroup (column 1), the impact on the primary enrollment outcome for the subgroup 
(column 2), the p-value denoting the likelihood that the impact could have been observed by chance (column 3), the impact on the primary 
completion outcome (column 4), and the associated p-value (column 5). The Difference rows report the difference in impact estimates between the 
two subgroups (for example, the impact for females subtracted from the impact for males; columns 2 and 4), and the p-value indicating whether 
this difference is statistically significantly different from zero (columns 3 and 5). Appendix B describes how these exploratory analyses were 
conducted. Exhibit B.10 provides details on the definitions of the subgroups reported. Percentages may not add up due to rounding. 


4 The expected family contribution (EFC), which is a measure of a family’s financial strength, is an index score calculated based on taxed and 
untaxed income, assets, and benefits (see https://studentaid.gov/help-center/answers/article/what-is-expected-family-contribution). 
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> An individual may qualify as a dislocated worker if they meet one of the following conditions: has lost a job, has been laid off or received a lay-off 
notice, is receiving unemployment benefits due to being laid off or losing a job and is unlikely to return to a previous occupation, is self-employed 
but unemployed due to economic conditions or a natural disaster, is the spouse of an active duty member of the armed forces and is unemployed 
or underemployed and is experiencing difficulty in obtaining or upgrading employment, or is a displaced homemaker (a person who previously 
provided unpaid services to the family, is no longer supported by the spouse, is unemployed or underemployed, and is having trouble finding or 
upgrading employment) (see https://fafsa.gov/fotw2021/help/fahelp69.htm). 


©To determine whether a program was in a high-demand occupation in its state, the study adapted criteria established by the U.S. Department of 
Labor’s Occupational Information Network (O*NET) for “Bright Outlook” occupations that were expected to grow rapidly or have large numbers of 
job openings between 2014 and 2016, midway through the study period. Additional details on how the study determined whether a program was in 
high demand are in Section B.3 of Appendix B. 


Exhibit C.4. Impacts on enrollment in and completion of a program at study schools for subgroups, 
Experiment 2 


Completed a program at the 
ialtehvacre sere) | 


Enrolled in the study school 


Male 816 15.3 0.000 11.4 0.000 
Female 1,431 14.8 0.000 8.5 0.000 
Difference 0.6 0.880 2.9 0.477 
Under 25 733 12.5 0.000 6.0 0.033 
Over 25 1,536 15.6 0.000 10.9 0.000 
Difference 3.1 0.271 -4.9 0.109 
EFC = 0? 538 9.0 0.012 5.2 0.184 
EFC > O 1,524 16.9 0.000 10.5 0.000 
Difference -7.9 0.059 5.4 0.234 
Dislocated worker? 468 16.4 0.000 8.4 0.045 
Not a dislocated worker 1,597 14.5 0.000 9.4 0.000 
Difference 1.9 0.670 -1.0 0.826 
High-demand occupation‘ 1,302 15.5 0.000 8.4 0.000 
Not a high-demand occupation 964 13.3 0.000 10.4 0.000 
Difference 2.2 0.403 -2.0 0.493 
County unemployment rate 1,901 15.0 0.000 10.4 0.000 
above national average 

County unemployment rate 369 12.9 0.000 3.8 0.289 
below national average 

Difference 2.1 0.545 6.5 0.076 
Enrolled prior to random 313 24.9 0.000 15.8 0.000 
assignment 

Not enrolled prior to random 1,957 13.0 0.000 8.2 0.000 
assignment 

Difference 11.9 0.001 7.6 0.053 


SOURCE: Random Assignment System, school records, Office of Federal Student Aid. 

NOTE: This exhibit reports the number of students in each subgroup (column 1), the impact on the primary enrollment outcome for the subgroup 
(column 2), the p-value denoting the likelihood that the impact could have been observed by chance (column 3), the impact on the primary 
completion outcome (column 4), and the associated p-value (column 5). The Difference rows report the difference in impact estimates between the 
two subgroups (for example, the impact for females subtracted from the impact for males; columns 2 and 4), and the p-value indicating whether 
this difference is statistically significantly different from zero (columns 3 and 5). Appendix B describes how these exploratory analyses were 
conducted. Appendix B, Exhibit B.10 provides details on the definitions of the subgroups reported. 

4 The expected family contribution (EFC), a measure of a family’s financial strength, is an index score calculated based on taxed and untaxed 
income, assets, and benefits (see https://studentaid.gov/help-center/answers/article/what-is-expected-family-contribution). 


> An individual may qualify as a dislocated worker if they meet one of the following conditions: has lost job, has been laid off or received a lay-off 
notice, is receiving unemployment benefits due to being laid off or losing a job and is unlikely to return to a previous occupation, is self-employed 
but unemployed due to economic conditions or a natural disaster, is the spouse of an active duty member of the armed forces and is unemployed 
or underemployed and is experiencing difficulty in obtaining or upgrading employment, or is a displaced homemaker (a person who previously 
provided unpaid services to the family, is no longer supported by the spouse, is unemployed or underemployed, and is having trouble finding or 
upgrading employment) (see https://fafsa.gov/fotw2021/help/fahelp69.htm). 


©To determine whether a program was in a high-demand occupation in its state, the study adapted criteria established by the U.S. Department of 
Labor’s Occupational Information Network (O*NET) for “Bright Outlook” occupations that were expected to grow rapidly or have large numbers of 
job openings between 2014 and 2016, midway through the study period. Additional details on how the study determined whether a program was in 
high demand are provided in Section B.3 of Appendix B. 
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C.3. Additional information supporting estimates of students most likely to use experimental 
Pell Grants 


In addition to knowing the share of students offered experimental Pell Grants who went on to use the 
funds as reported earlier, it is important to understand why students decided to use the funds by 
enrolling in a program at study schools. To shed some light on this, the study examined what student 
characteristics were correlated with the decision to use experimental Pell Grants for those offered 
the funds. The sign of the coefficients reported in Exhibit C.5 denote whether there was a positive or 
negative relationship between each student characteristic and the decision to use experimental Pell 
Grants, and the magnitude of the coefficient signifies how much the probability that a student used 
experimental Pell Grants changes given a one-unit shift in each student characteristic, holding the 
other characteristics constant. Exhibit C.5 also reports p-values which signify how likely it is that the 
relationship between the student characteristic and the decision to use experimental Pell Grants is 
due to chance. 


Of the characteristics analyzed (see Appendix B, Exhibit B.7 and Section B.4 for an explanation of the 
characteristics chosen and details on the analysis method), for students in Experiment 1, only age 
had a correlation with the decision to take up the offer of experimental Pell that was significantly 
different from zero (that is, unlikely to be observed by chance). Older students were significantly less 
likely to take up experimental Pell, compared to younger students (Exhibit C.5). For students in 
Experiment 2, only one student characteristic had a correlation with the decision to take up the offer 
of experimental Pell that was significantly different from zero. Students with a larger expected family 
contribution (EFC; that is, those who were expected to pay more out-of-pocket for attending school 
due to having higher incomes or greater assets) were significantly less likely to take up the offer of 
experimental Pell, compared to students with a smaller EFC (Exhibit C.5). 
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Exhibit C.5. Correlation between student characteristics and the decision to take up the offer of 
experimental Pell Grants, Experiment 1 


Student characteristic 


Experiment 1 


Experiment 2 


Age (years) -0.01 0.009 0.0008 0.468 
Gender (female) 0.04 0.525 -0.02 0.458 
High school diploma or equivalent -0.03 0.859 0.18 0.068 
Expected family contribution (EFC)? 0.000009 0.364 -0.00002 0.000 
Indicator for EFC = 0 -0.07 0.385 0.04 0.157 
Dislocated worker? 0.10 0.197 -0.02 0.393 
Regression statistics 

N 254 - 1,363 - 
R-squared 0.42 - 0.38 - 


SOURCE: Random Assignment System, school records, FSA. 
NOTE: Appendix B, Section B.4 describes how this exploratory analysis was conducted. Appendix B, Exhibit B.7 provides details on the student 


characteristics reported in this exhibit. 


4 The expected family contribution, a measure of a family’s financial strength, is an index score calculated based on taxed and untaxed income, 
assets, and benefits (see https://studentaid.gov/help-center/answers/article/what-is-expected-family-contribution). 


> An individual may qualify as a dislocated worker if they meet one of the following conditions: has lost job, has been laid off or received a lay-off 
notice, is receiving unemployment benefits due to being laid off or losing a job and is unlikely to return to a previous occupation, is self-employed 
but unemployed due to economic conditions or a natural disaster, is the spouse of an active duty member of the Armed Forces and is unemployed 
or underemployed and is experiencing difficulty in obtaining or upgrading employment, or is a displaced homemaker (a person who previously 
provided unpaid services to the family, is no longer supported by the spouse, is unemployed or underemployed, and is having trouble finding or 
upgrading employment) (see https://fafsa.gov/fotw2021/help/fahelp69.htm). 


C.4. Additional information supporting assessment of programs students expressed interest in, 


enrolled in, and completed 


This section reports on the percentage of students in each experiment who expressed interest in, 
enrolled in, and completed different types of programs. Programs are classified by 2-digit 
Classification of Instructional Program (CIP) codes.*4 Findings are described earlier in the report. 


Exhibit C.6. Programs students expressed interest in at random assignment, Experiment 1 


Percentage of Experiment 1 students 


Offered experimental 


Not offered 
experimental Pell Grant Overall 


Program Pell Grant 

Health professions and related 5.5 10.0 oP) 
programs 

Business, management, marketing, and 6.7 6.9 6.8 
related support services 

Other 3.5 1.9 2.9 
Missing CIP code 84.3 81.3 83.1 
Total number of students 254 160 414 


SOURCE: Random Assignment System, National Center for Education Statistics (https://nces.ed.gov/pubs2002/cip2000/). 


NOTE: This exhibit shows the programs students expressed interest in at the time of random assignment and reports the two-digit Classification of 
Instructional Programs (CIP) code label for each program; also reported are the percentages of students overall, of students who were offered 
experimental Pell Grants, and of students who were not offered experimental Pell Grants who expressed interest in a program. “Other” includes 
multiple programs that have been combined to protect respondent confidentiality in accordance with National Center for Education Statistics 


standards. 


24 https://nces.ed.gov/pubs2002/cip2000/ 
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Exhibit C.7. Programs students expressed interest in at random assignment, Experiment 2 


Program 


Percentage of Experiment 2 students 


Offered experimental 
Pell Grant 


INColmoyaiearuere | 
experimental 
Pell Grant 


Transportation and materials moving 46.8 46.3 46.6 
Health professions and related programs 34.5 34.8 34.6 
Homeland security, law enforcement, 
firefighting and related protective 7.8 7.5 7.7 
services 
Mechanic and repair 
technologies/technicians ie fe ie 
Business, management, marketing, and 

: 1.1 1.5 1.3 
related support services 
Precision production 1.0 0.6 0.8 
Family and consumer sciences/human 0.7 0.9 0.8 
sciences 
Other or missing CIP code 0.7 1.2 0.9 
Total number of students 1,363 907 2,270 


SOURCE: Random Assignment System, National Center for Education Statistics (https://nces.ed.gov/pubs2002/cip2000/). 


NOTE: This exhibit shows the programs students expressed interest in at the time of random assignment and reports the two-digit Classification of 
Instructional Programs (CIP) code label for each program; also reported are the percentages of students overall, of students who were offered 
experimental Pell Grants, and of students who were not offered experimental Pell Grants who expressed interest in a program. “Other” programs 
are those with fewer than 10 students per program overall. “Other” includes multiple programs that have been combined with missing CIP codes to 
protect respondent confidentiality in accordance with National Center for Education Statistics standards. 


Exhibit C.8. Programs students enrolled in, Experiment 1 


Percentage of Experiment 1 students 


Not offered 
Offered experimental experimental 

Program Pell Grant Pell Grant 
Health professions and related programs 39.0 37.5 38.4 
Business, management, marketing, and 71 awh 6.0 
related support services ; ; ; 
Mechanic and repair 
technologies/technicians Be of ae 
Education 3.9 1.9 3.1 
Engineering technologies and 28 19 24 
engineering-related fields : . ; 
Other or missing CIP code 16.5 6.3 12.6 
Did not enroll in a study school 24.4 48.1 33.6 
Total number of students 254 160 414 


SOURCE: School records, National Center for Education Statistics (https://nces.ed.gov/pubs2002/cip2000/). 


NOTE: This exhibit shows the programs students enrolled in and reports the two-digit Classification of Instructional Programs (CIP) code label for 
each program; also shown are the percentages of students overall who enrolled in each program, as well as percentages of enrolled students who 
were offered experimental Pell Grants and percentages of those enrolled who were not offered experimental Pell Grants. “Other” includes multiple 
programs that have been combined to protect respondent confidentiality in accordance with National Center for Education Statistics standards. 
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Exhibit C.9. Programs students enrolled in, Experiment 2 


Program 


Offered experimental 


Pell Grant 


Percentage of Experiment 2 students 


INColmoyaieruere | 
experimental 
Pell Grant 


Transportation and materials moving 35.1 28.6 32.5 
Health professions and related programs 20.3 14.7 18.1 
Construction trades 4.7 2.0 3.6 
Homeland security, law enforcement, 4.4 3.3 4.0 
firefighting and related protective 

services 

Precision production 0.9 0.7 0.8 
Mechanic and repair 0.6 0.4 0.5 
technologies/technicians 

Family and consumer sciences/human 0.5 0.6 0.5 
sciences 

Business, management, marketing, and 0.2 0.3 0.3 
related support services 

Other or missing CIP code 0.9 1.3 1.1 
Did not enroll in a study school 32.4 48.2 38.7 
Total number of students 1,363 907 2,270 


SOURCE: School records, National Center for Education Statistics (https://nces.ed.gov/pubs2002/cip2000/). 


NOTE: This exhibit shows the programs students enrolled in and reports the two-digit Classification of Instructional Programs (CIP) code label for 
each program; also shown are the percentages of students overall who enrolled in each program, as well as percentages of enrolled students who 


were offered experimental Pell Grants and percentages of those enrolled who were not offered experimental Pell Grants. “Other” includes multiple 
programs that have been combined to protect respondent confidentiality in accordance with National Center for Education Statistics standards. 


Exhibit C.10. Programs students completed, Experiment 1 


Percentage of Experiment 1 students 


Not offered 
Offered experimental experimental 

Program Pell Grant Pell Grant 
Health professions and related programs 29.1 28.8 29.0 
Mechanic and repair 4.3 0.0 2.7 
technologies/technicians 
Other 17.7 6.9 13.5 
Did not complete a program in a study 48.8 64.4 54.8 
school 
Total number of students 254 160 414 


SOURCE: School records, National Center for Education Statistics (https://nces.ed.gov/pubs2002/cip2000/). 

NOTE: This exhibit shows the programs students completed and reports the two-digit Classification of Instructional Programs (CIP) code label for 
each program; also shown are the percentages of students overall, of students who were offered experimental Pell Grants, and of students who 
were not offered experimental Pell Grants who completed a program. “Other” includes multiple programs that have been combined to protect 
respondent confidentiality in accordance with National Center for Education Statistics standards. 
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Exhibit C.11. Programs students completed, Experiment 2 


Program 


Percentage of Experiment 2 students 


Offered experimental 
Pell Grant 


INColmoyscaluere | 
experimental 
Pell Grant 


Overall 


Transportation and materials moving 26.1 23.2 24.9 
Health professions and related programs 13.9 8.8 11.9 
Homeland security, law enforcement, 3.5 2.8 3.2 
firefighting and related protective services 

Construction trades 3.2 1.8 2.6 
Precision production 0.4 0.3 0.4 
Other or missing CIP code 1.8 1.4 1.7 
Did not complete a program in a study school 51.9 62.3 56.0 
Total number of students 1,363 907 2,270 


SOURCE: School records, National Center for Education Statistics (https://nces.ed.gov/pubs2002/cip2000/). 
NOTE: This exhibit shows the programs students completed and reports the two-digit Classification of Instructional Programs (CIP) code label for 
each program; also reported are the percentages of students overall, of students who were offered experimental Pell Grants, and of students who 
were not offered experimental Pell Grants who completed a program. “Other” includes multiple programs that have been combined to protect 

respondent confidentiality in accordance with National Center for Education Statistics standards. 


Exhibit C.12. High-demand programs that students completed, Experiment 1 


Percentage of Experiment 1 students 


INolmeyitauctel 
Offered experimental experimental 
Program Pell Grant Pell Grant Overall 
Health professions and related programs 26.0 23.8 25.1 
Other 12.5 5.0 9.6 
Did not complete a high-demand program 59.1 68.8 62.8 
Missing CIP code or high-demand indicator 2.4 2.5 2.4 
Total number of students 254 160 414 


SOURCE: School records, O*Net (https://www.onetonline.org/find/bright), National Center for Education Statistics 
(https://nces.ed.gov/pubs2002/cip2000/). 

NOTE: This exhibit contains information on the high-demand programs students completed and reports the two-digit Classification of Instructional 
Programs (CIP) code label for each program; also reported are the percentages of students overall, of students who were offered experimental Pell 


Grants and completed a program and of those who were not offered experimental Pell Grants and completed a program. . “Other” includes 
multiple programs that have been combined to protect respondent confidentiality in accordance with National Center for Education Statistics 
standards. Appendix B, Section B.5 describes how the study identified high-demand programs. 
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Exhibit C.13. High-demand programs that students completed, Experiment 2 


Percentage of Experiment 2 students 


Not offered 

Offered experimental experimental 
Program Pell Grant Pell Grant Overall 
Transportation and materials moving 25.9 23.2 24.8 
Health professions and related programs 10.8 6.6 9.1 
Construction trades 3.2 1.8 2.6 
Other or missing CIP code or high-demand 2.2 1.4 1.9 
indicator 
Did not complete a high-demand program 57.8 67.0 61.5 
Total number of students 1,363 907 2,270 


SOURCE: Random Assignment System, O*Net (https://www.onetonline.org/find/bright), National Center for Education Statistics 


(https://nces.ed.gov/pubs2002/cip2000/). 


NOTE: This exhibit shows the high-demand programs students completed and reports the two-digit Classification of Instructional Programs (CIP) 
code label for each program; also reported are the percentages of students overall, of students who were offered experimental Pell Grants, and of 
students who were not offered experimental Pell Grants who completed each program. . “Other” includes multiple programs that have been 
combined with missing CIP codes to protect respondent confidentiality in accordance with National Center for Education Statistics standards. 


Appendix B, Section B.5 describes how the study identified high-demand programs. 


C.5. Additional information supporting estimates of federal student loan use and disbursement 


amounts 


This section reports on the number and percentage of students offered experimental Pell Grants in 
each experiment who used federal student loans during the study period (2012-2018), as well as the 
average federal loan amount disbursed per student during the study period. 


Exhibit C.14. Federal student loan take-up rates and disbursement amounts for those offered 


experimental Pell Grants 


Experiment 1 Experiment 2 


Number of students offered experimental Pell Grants 254 1,363 
Number of students offered experimental Pell Grants who took out 87 126 
federal student loans 

Federal student loan take-up rate (number of students who took out 34% 9% 
student loans divided by the number of students offered experimental 

Pell Grants) 

Average federal student loan amount disbursed (total amount disbursed $12,950 $4,021 
divided by the number of students who took out student loans) 

Number of students offered experimental Pell Grants who used one 170 705 
Number of students who used experimental Pell Grants and also took 63 39 
out federal student loans 

Share of students who used experimental Pell Grants who took out 37% 6% 


federal student loans (number of students who used experimental Pell 
Grants who also took out student loans divided by the number of 
students who used experimental Pell Grants) 


Source: School records, Office of Federal Student Aid. 
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C.6. Additional information supporting the estimated cost of the 2011 Pell Grant Experiments if 


they were made official policy 


This section reports the statistics the study used to estimate the potential cost if the eligibility 
expansions that were piloted in the 2011 Pell Grant Experiments were made official policy. The 
potential cost was obtained by multiplying the estimated number of students eligible for each waiver by 
the average size of the Pell Grant disbursed under each experiment. 


Exhibit C.15. Estimate of what the 2011 Pell Grant Experiments would cost if they were made 


official policy 

Statistic Experiment 1 Experiment 2 

1. Total number of undergraduate students in U.S.? 23,620,218 23,620,218 

2. Percentage (%) of undergraduate students not in a degree program? 3 3 

3. Number of undergraduate students not in a degree program‘ 708,607 708,607 

4. Percentage (%) of undergraduate students in U.S. in a non-degree 54.6 54.6 
program with EFC < $5,1984 

5. Number of undergraduate students not in a degree program with 386,899 386,899 
EFC < $5,198° 

6. Percentage (%) of undergraduate students in certificate programs 15 NA 
who already have a bachelor's degree’ 

7. Number of undergraduate students who already have a bachelor's 58,035 NA 
degree, who are not in a degree program, and who have EFC < 
$5,1988 

8. Average size of experimental Pell Grant" $3,556 $1,312 
Potential costi $206,372,018 $507,611,712 


SOURCE: Various; see below. 

NOTES: See table footnotes below for how each numbered table row contributed to the calculation. 
@U.S. Department of Education, National Center for Education Statistics, IPEDS, Fall 2015, 12-Month Enrollment component (final data). 
>U.S. Department of Education, National Center for Education Statistics, 2015-16 National Postsecondary Student Aid Study (NPSAS:16). 


©Calculated: row 1 multiplied by row 2. 


4U.S. Department of Education, National Center for Education Statistics, 2015-16 National Postsecondary Student Aid Study (NPSAS:16). $5,198 is the 
maximum EFC allowed to be Pell Grant-eligible in 2015 - 2016. (Source: DJA Financial Aid Services, https://gotodja.com/2015/2015-2016-federal-pell- 
grant-payment-and-disbursement- 
schedules/#:~:text=The%202015%2D2016%20maximum%20Pell%20Grant%20award%20is%20%245%2C775.&text=The%20corresponding%20maxi 
mum%20Pell%20Grant,scheduled%20award%20amount%20is%20%24588). 


°Calculated: row 3 multiplied by row 4. 


‘Source: Carnevale, A. P., Rose, S. J., & Hanson, A. R. (2012). Certificates: Gateway to gainful employment and college degrees, 
https://cew.georgetown.edu/wp-content/uploads/2014/11/Certificates.FullReport.061812.pdf. NA = not applicable for Experiment 2. 


® Calculated: row 5 multiplied by row 6. NA = not applicable for Experiment 2. 

Source: School records. 

‘Calculated: row 5 multiplied by row 8 (Experiment 1), row 7 multiplied by row 8 (Experiment 2). 
EFC = expected family contribution. 
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APPENDIX D. SENSITIVITY ANALYSES 


This appendix describes tests of whether the effects described in the report were sensitive to certain 
analytic decisions made by the study team. Researchers frequently conduct sensitivity testing to 
determine how confident they can be about their key findings. The sensitivity analyses here 
examined whether the study results would differ by: 


Changing the sample of students used to calculate the impacts on the primary enrollment and 
completion outcomes (sensitivity analyses 1-4), and 


Using different methods for treating missing covariate data and estimating the impacts of 
experimental Pell Grants on the primary outcomes (sensitivity analyses 5 and 6). 


The description and purpose of the sensitivity analyses are described in Exhibit D.1. 


The results of the sensitivity analyses for both experiments, shown in Exhibits D.2 for Experiment 1 
and D.3 for Experiment 2, were consistent with findings reported in the main body of the report.”° 


Exhibit D.1. Description and purpose of sensitivity analyses 


Sensitivity analysis Purpose 


Changing the sample of students 


1. 


Include students who enrolled in both experiments. 
The main analysis excludes students who enrolled in 
both experiments to isolate the impact of each 
experiment separately. This sensitivity analysis 
includes those students and adds an indicator for 
enrollment in both experiments to the benchmark 
model. 


To explore whether results are sensitive to including these 
participants. In addition, since statistical power to detect effects 
in this study is modest, particularly for Experiment 1, this 
sensitivity analysis takes advantage of the increased sample size 
and statistical power afforded by adding these. Including 
individuals who enrolled in both experiments increases the 
analysis sample size and allows the study to test whether results 
are sensitive to including these individuals. 


2. Include students who were subject to an error in the To explore whether results are sensitive to including these 
Random Assignment System. The main analysis students, and to take advantage of the increased sample size 
excludes students who were subject to an error inthe | and statistical power afforded by adding these students to the 
Random Assignment System (see Appendix B for a analysis. 
description of these errors). This sensitivity analysis 
includes them in the analysis sample. 

3. Exclude students with an enrollment date before the To explore whether results are sensitive to excluding these 


random assignment date. Some students were 
already enrolled at study schools before the 
experiments began. (The Office of Federal Student 
Aid permitted study schools to include students in the 
experiments who were already enrolled at the 
schools.) The main analysis includes these students. 
This sensitivity analysis estimates impacts for the 
sample of students whose program enrollment dates 
occur on or after their random assignment dates. 


students. 


25 The model specifications are the same as for the main impact analyses described in Appendix B. 


6 One exception is in Experiment 1, sensitivity analysis 4: The impact on completion of a program at the study school was 
positive but not significantly different from zero (Exhibit D.2) when participants who were found ineligible for Pell Grants after 
random assignment were excluded from the analytic sample. 
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Sensitivity analysis Purpose 


4. 


Exclude participants who were found ineligible for 
Pell Grants after random assignment. Some students 
were found to be ineligible for Pell Grants after 
random assignment ended and after the Random 
Assignment System was closed. The main analysis 
includes these students. This sensitivity analysis 
excludes them. 


To explore whether results are sensitive to excluding these 
students. 


Using different analysis methods 


5. 


Use casewise deletion to account for missing 
covariate data. The main analysis imputes missing 
data using a method described in Appendix B. This 
sensitivity analysis estimates the impact model 
without imputing any missing covariate data. In this 
analysis, students with missing covariate data are 
dropped from the analysis. 


To explore whether results are sensitive to the method of 
accounting for missing data in the benchmark approach. 


Use an alternative estimation strategy for binary 
outcomes. The study’s primary outcomes (enrollment 
in the study school and completion of a program at 
the study school) are binary. The main analysis uses 
ordinary least squares (OLS) to estimate the impacts 
on these outcomes in a linear probability model. This 
sensitivity analysis uses a nonlinear logit model. 


To explore whether impact results are sensitive to the model 
used to estimate impacts on binary outcomes. When predicting 
the probability that an individual enrolled in a study school 
using parameters estimated in a linear OLS model, it is possible 
for the predicted probabilities to be greater than one or less 
than zero. Logit models do not share this weakness—using logit 
models, predicted probabilities always fall between zero and 
one. 
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Exhibit D.2. Sensitivity analyses for impacts on primary outcomes for Experiment 1 


(Oitoyaet Not offered 
experimental experimental 

Pell Grant Pell Grant 
Outcome (percentage) Impact | p-value 
Primary 
Enrolled in the study school 77.9 254 51.9 160 26.0 0.000 
Completed a program at the study school 52.4 254 35.6 160 16.7 0.000 
Sensitivity analysis 1: Include students who enrolled in both experiments 
Enrolled in the study school 76.1 301 52.7 184 23.4 0.000 
Completed a program at the study school 50.9 301 36.4 184 14.4 0.001 
Sensitivity analysis 2: Include students who were subject to an error in the Random Assignment System 
Enrolled in the study school 75.8 274 51.9 160 23.9 0.000 
Completed a program at the study school 52.1 274 35.6 160 16.5 0.000 
Sensitivity analysis 3: Exclude students with an enrollment date before the random assignment date 
Enrolled in the study school 70.3 191 39.3 122 31.0 0.000 
Completed a program at the study school 50.9 191 26.2 122 24.7 0.000 


Sensitivity analysis 4: Exclude participants who were found ineligible for experimental Pell funds after random 
assignment 


Enrolled in the study school 77.8 194 60.7 122 17.2 0.000 
Completed a program at the study school 49.0 194 41.8 122 7.2 0.143 
Sensitivity analysis 5: Use casewise deletion to account for missing covariate data 

Enrolled in the study school 73.6 208 51.9 127 21.7 0.000 
Completed a program at the study school 46.4 208 35.6 127 10.8 0.026 
Sensitivity analysis 6: Use an alternative estimation strategy for binary outcomes 

Enrolled in the study school 78.4 208 59.1 127 19.3 0.000 
Completed a program at the study school 51.4 208 40.9 127 10.5 0.022 


SOURCE: Random Assignment System, school records, Office of Federal Student Aid. 


NOTE: Sensitivity analyses 1-4 use the same method as described in Appendix B to estimate the impacts of the experiments on the primary 
outcomes. Sensitivity analysis 5 uses covariates with missing data instead of the method described in Appendix B. Sensitivity analysis 6 uses a 
nonlinear logit model and covariates with missing data and excludes school and program fixed effects. Column 5 contains impact estimates that are 
regression-adjusted for student socioeconomic characteristics measured before random assignment, as described in Appendix B. The “Offered 
experimental Pell Grant” group mean (column 1) equals the sum of the unadjusted control group mean (column 3) and the regression-adjusted 
impact estimate (column 5). Column 2 contains the number of students in the “Offered experimental Pell Grant” group who were included in the 
sensitivity analysis, and column 4 contains the number of students in the “Not offered experimental Pell Grant” group who were included in the 
sensitivity analysis. Column 6 reports the probability that the impact in column 5 could have been observed by chance, with small p-values 
indicating a low likelihood that the impact is due to chance and not to the offer of experimental Pell funds. Percentages may not add up due to 
rounding. 
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Exhibit D.3. Sensitivity analyses for impacts on primary outcomes for Experiment 2 


(itoysel Not offered 
experimental Pell experimental Pell 
Grant Grant 
Outcome (percentage) Impact | p-value 
Primary 
Enrolled in the study school 66.4 1,363 51.8 907 14.6 0.000 
Completed a program at the study school 47.0 1,363 37.7 907 9.3 0.000 
Sensitivity analysis 1: Include students who enrolled in both experiments 
Enrolled in the study school 66.4 1,410 52.0 931 14.4 0.000 
Completed a program at the study school 46.8 1,410 37.8 931 9.0 0.000 
Sensitivity analysis 2: Include students who received incorrect assignments 
Enrolled in the study school 66.3 1,484 51.7 910 14.6 0.000 
Completed a program at the study school 47.2 1,484 37.6 910 9.6 0.000 
Sensitivity analysis 3: Exclude students with an enrollment date before the random assignment date 
Enrolled in the study school 62.2 1,175 45.1 782 17.1 0.000 
Completed a program at the study school 44.1 1,175 33.1 782 11.0 0.000 


Sensitivity analysis 4: Exclude participants who were found ineligible for experimental Pell funds after random 
assignment 


Enrolled in the study school 68.4 1,201 53.0 793 15.4 0.000 
Completed a program at the study school 47.8 1,201 38.1 793 9.7 0.000 
Sensitivity analysis 5: Use casewise deletion to account for missing covariate data 

Enrolled in the study school 66.7 1,233 51.8 827 14.9 0.000 
Completed a program at the study school 46.9 1,233 37.7 827 9.1 0.000 
Sensitivity analysis 6: Use an alternative estimation strategy for binary outcomes 

Enrolled in the study school 68.0 1,233 52.8 827 15.2 0.000 
Completed a program at the study school 47.8 1,233 38.3 827 9.5 0.000 


SOURCE: Random Assignment System, school records, Office of Federal Student Aid. 

NOTE: Sensitivity analyses 1-4 use the same method described in Appendix B to estimate the impacts of the experiments on the primary outcomes. 
Sensitivity analysis 5 uses covariates with missing data instead of the method described in Appendix B. Sensitivity analysis 6 uses a nonlinear logit 
model and covariates with missing data and excludes school and program fixed effects. Column 5 contains impact estimates that are regression- 
adjusted for student socioeconomic characteristics measured before random assignment, as described in Appendix B. The “Offered experimental 
Pell Grant” group mean (column 1) equals the sum of the unadjusted control group mean (column 3) and the regression-adjusted impact estimate 
(column 5). Column 2 contains the number of students in the “Offered experimental Pell Grant” group who were included in the sensitivity 
analysis, and column 4 contains the number of students in the “Not offered experimental Pell Grant” group who were included in the sensitivity 
analysis. Column 6 reports the probability that the impact in column 5 could have been observed by chance, with small p-values indicating a low 
likelihood that the impact is due to chance and not to the offer of experimental Pell funds. Percentages may not add up due to rounding. 
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